Search

We study offline multitask representation learning in reinforcement learning (RL), where a learner is provided with an offline dataset from different tasks that share a common representation and is asked to learn the shared representation. We theoretically investigate offline multitask low-rank RL, and propose a new algorithm called MORL for offline multitask representation learning. Furthermore, we examine downstream RL in reward-free, offline and online scenarios, where a new task is introduced to the agent that shares the same representation as the upstream offline tasks. Our theoretical results demonstrate the benefits of using the learned representation from the upstream offline task instead of directly learning the representation of the low-rank model.

The problem of two-player zero-sum Markov games has recently attracted increasing interests in theoretical studies of multi-agent reinforcement learning (RL). In particular, for finite-horizon episodic Markov decision processes (MDPs), it has been shown that model-based algorithms can find an $\epsilon$-optimal Nash Equilibrium (NE) with the sample complexity of $O(H^3SAB/\epsilon^2)$, which is optimal in the dependence of the horizon $H$ and the number of states $S$ (where $A$ and $B$ denote the number of actions of the two players, respectively). However, none of the existing model-free algorithms can achieve such an optimality. In this work, we propose a model-free stage-based Q-learning algorithm and show that it achieves the same sample complexity as the best model-based algorithm, and hence for the first time demonstrate that model-free algorithms can enjoy the same optimality in the $H$ dependence as model-based algorithms. The main improvement of the dependency on $H$ arises by leveraging the popular variance reduction technique based on the reference-advantage decomposition previously used only for single-agent RL. However, such a technique relies on a critical monotonicity property of the value function, which does not hold in Markov games due to the update of the policy via the coarse correlated equilibrium (CCE) oracle. Thus, to extend such a technique to Markov games, our algorithm features a key novel design of updating the reference value functions as the pair of optimistic and pessimistic value functions whose value difference is the smallest in the history in order to achieve the desired improvement in the sample efficiency.

General function approximation is a powerful tool to handle large state and action spaces in a broad range of reinforcement learning (RL) scenarios. However, theoretical understanding of non-stationary MDPs with general function approximation is still limited. In this paper, we make the first such an attempt. We first propose a new complexity metric called dynamic Bellman Eluder (DBE) dimension for non-stationary MDPs, which subsumes majority of existing tractable RL problems in static MDPs as well as non-stationary MDPs. Based on the proposed complexity metric, we propose a novel confidence-set based model-free algorithm called SW-OPEA, which features a sliding window mechanism and a new confidence set design for non-stationary MDPs. We then establish an upper bound on the dynamic regret for the proposed algorithm, and show that SW-OPEA is provably efficient as long as the variation budget is not significantly large. We further demonstrate via examples of non-stationary linear and tabular MDPs that our algorithm performs better in small variation budget scenario than the existing UCB-type algorithms. To the best of our knowledge, this is the first dynamic regret analysis in non-stationary MDPs with general function approximation., Comment: ICML 2023

As representation learning becomes a powerful technique to reduce sample complexity in reinforcement learning (RL) in practice, theoretical understanding of its advantage is still limited. In this paper, we theoretically characterize the benefit of representation learning under the low-rank Markov decision process (MDP) model. We first study multitask low-rank RL (as upstream training), where all tasks share a common representation, and propose a new multitask reward-free algorithm called REFUEL. REFUEL learns both the transition kernel and the near-optimal policy for each task, and outputs a well-learned representation for downstream tasks. Our result demonstrates that multitask representation learning is provably more sample-efficient than learning each task individually, as long as the total number of tasks is above a certain threshold. We then study the downstream RL in both online and offline settings, where the agent is assigned with a new task sharing the same representation as the upstream tasks. For both online and offline settings, we develop a sample-efficient algorithm, and show that it finds a near-optimal policy with the suboptimality gap bounded by the sum of the estimation error of the learned representation in upstream and a vanishing term as the number of downstream samples becomes large. Our downstream results of online and offline RL further capture the benefit of employing the learned representation from upstream as opposed to learning the representation of the low-rank model directly. To the best of our knowledge, this is the first theoretical study that characterizes the benefit of representation learning in exploration-based reward-free multitask RL for both upstream and downstream tasks., Comment: Submitted for publication

In this paper, we consider a status updating system where updates are generated at a constant rate at $K$ sources and sent to the corresponding recipients through a noise-free broadcast channel. We assume that perfect channel state information (CSI) is available at the transmitter before each transmission, and the transmitter is able to utilize the CSI information to precode the updates. Our object is to design optimal precoding schemes to minimize the summed average \emph{age of information} (AoI) at the recipients. Under various assumptions on the size of each update $B$, the number of transmit antennas $M$, and the number of receive antennas $N$ at each user, this paper identifies the corresponding age-optimal precoding and transmission scheduling strategies. Specifically, for the case when $N=1$, a round-robin based updating scheme is shown to be optimal. For the two-user systems with $N>B$ or $M\notin[N:2N]$, framed updating schemes are proven to be optimal. For other cases in the two-user systems, a framed alternating updating scheme is proven to be $2$-optimal.

In this paper, we aim to establish the connection between Age of Information (AoI) in network theory, information uncertainty in information theory, and detection delay in time series analysis. We consider a dynamic system whose state changes at discrete time points, and a state change won't be detected until an update generated after the change point is delivered to the destination for the first time. We introduce an information theoretic metric to measure the information freshness at the destination, and name it as generalized Age of Information (GAoI). We show that under any state-independent online updating policy, if the underlying state of the system evolves according to a stationary Markov chain, the GAoI is proportional to the AoI. Besides, the accumulative GAoI and AoI are proportional to the expected accumulative detection delay of all changes points over a period of time. Thus, any (G)AoI-optimal state-independent updating policy equivalently minimizes the corresponding expected change point detection delay, which validates the fundamental role of (G)AoI in real-time status monitoring. Besides, we also investigate a Bayesian change point detection scenario where the underlying state evolution is not stationary. Although AoI is no longer related to detection delay explicitly, we show that the accumulative GAoI is still an affine function of the expected detection delay, which indicates the versatility of GAoI in capturing information freshness in dynamic systems., Comment: 6 pages, 7 figures

In this paper, we investigate the impact of coding on the Age of Information (AoI) in a two-user broadcast symbol erasure channel with feedback. We assume each update consists of $K$ symbols and the source is able to broadcast one symbol in each time slot. Due to random channel noise, the intended symbol at each user will be erased according to an independent and identically distributed (i.i.d.) Bernoulli process. A user is able to successfully decode an update if it accumulates sufficient information and successfully decodes the $K$ symbols of the update. Assuming prefect feedback information at the source right after the transmission of each symbol, our objective is to design an adaptive coding scheme to achieve small AoI at both users. We propose a novel coding scheme to judiciously combine symbols from different updates together, and analyze the AoI at both users. Compared with a baseline greedy scheme, the proposed adaptive coding scheme improves the AoI at the weak user by orders of magnitude without compromising the AoI at the strong user.

In this paper, we examine a status updating system where updates generated by the source are sent to the monitor through an erasure channel. We assume each update consists of $k$ symbols and the symbol erasure in each time slot follows an independent and identically distributed (i.i.d.) Bernoulli process. We assume rateless coding scheme is adopted at the transmitter and an update can be successfully decoded if $k$ coded symbols are received successfully. We assume perfect feedback available at the source, so that it knows whether a transmitted symbol has been erased instantly. Then, at the beginning of each time slot, the source has the choice to start transmitting a new update, or continue with the transmission of the previous update if it is not delivered yet. We adopt the metric "Age of Information" (AoI) to measure the freshness of information at the destination, where the AoI is defined as the age of the latest decoded update at the destination. Our objective is to design an optimal online transmission scheme to minimize the time-average AoI. The transmission decision is based on the instantaneous AoI, the age of the update being transmitted, as well as the number of successfully delivered symbols of the update. We formulate the problem as a Markov Decision Process (MDP) and identify the monotonic threshold structure of the optimal policy. Numerical results corroborate the structural properties of the optimal solution., Comment: To appear in ICC 2019

In this paper, we consider a scenario where a source continuously monitors an object and sends time-stamped status updates to a destination through a rate-limited link. We assume updates arrive randomly at the source according to a Bernoulli process. Due to the link capacity constraint, it takes multiple time slots for the source to complete the transmission of an update. Therefore, when a new update arrives at the source during the transmission of another update, the source needs to decide whether to skip the new arrival or to switch to it, in order to minimize the expected average age of information (AoI) at the destination. We start with the setting where all updates are of the same size, and prove that within a broadly defined class of online policies, the optimal policy should be a renewal policy, and has a sequential switching property. We then show that the optimal decision of the source in any time slot has threshold structures, and only depends on the age of the update being transmitted and the AoI at the destination. We then consider the setting where updates are of different sizes, and show that the optimal Markovian policy also has a multiple-threshold structure. For each of the settings, we explicitly identify the thresholds by formulating the problem as a Markov Decision Process (MDP), and solve it through value iteration. Special structural properties of the corresponding optimal policy are utilized to reduce the computational complexity of the value iteration algorithm., Comment: 13 pages, 6 figures. arXiv admin note: substantial text overlap with arXiv:1806.08698

Consider an energy harvesting (EH) sensor that continuously monitors a system and sends time-stamped status update to a destination. The sensor harvests energy from nature and uses it to power its updating operations. The destination keeps track of the system status through the successfully received updates. With the recently introduced information freshness metric "Age of Information" (AoI), our objective is to design optimal online status updating policy to minimize the long-term average AoI at the destination, subject to the energy causality constraint at the sensor. Due to the noisy channel between the sensor and the destination, each transmitted update may be erased with a fixed probability, and the AoI at the destination will be reset to zero only when an update is successfully received. We first consider status updating without feedback available to the sensor and show that the Best-effort Uniform updating (BU) policy is optimal. We then investigate status updating with perfect feedback to the sensor and prove the optimality of the Best-effort Uniform updating with Retransmission (BUR) policy. In order to prove the optimality of the proposed policies, for each case, we first identify a lower bound on the long-term average AoI among a broad class of online policies, and then construct a sequence of virtual policies to approach the lower bound asymptotically. Since those virtual policies are sub-optimal to the original policy, the original policy is thus optimal., Comment: arXiv admin note: substantial text overlap with arXiv:1803.01462

Consider a scenario where a source continuously monitors an object and sends time-stamped status updates to a destination through a rate-limited link. In order to measure the "freshness" of the status information available at the destination, we adopt the metric called Age of Information (AoI). We assume all updates are of the same size, and arrive randomly at the source according to a Bernoulli process. Due to the link capacity constraint, it takes $d$ ($d\geq 2$) time slots for the source to complete the transmission of an update. Therefore, when a new update when arrives at the source and it is transmitting another update, the source needs to decide whether to skip the new arrival or to switch to it, in order to minimize the expected average AoI at the destination. We prove that within a broadly defined class of online policies, the optimal policy should be a renewal policy, and has a sequential switching property. We then show that the optimal decision the source should take in any time slot has a multiple-threshold structure, and only depends on the age of the update being transmitted and the AoI in the system. The thresholds are then numerically identified by formulating the problem as a Markov Decision Process (MDP)., Comment: 5 pages, 4 figures

Consider an energy harvesting sensor continuously monitors a system and sends time-stamped status update to a destination. The destination keeps track of the system status through the received updates. Under the energy causality constraint at the sensor, our objective is to design an optimal online status updating policy to minimize the long-term average Age of Information (AoI) at the destination. We focus on the scenario where the the channel between the source and the destination is noisy, and each transmitted update may fail independently with a constant probability. We assume there is no channel state information or transmission feedback available to the sensor. We prove that within a broadly defined class of online policies, the best-effort uniform updating policy, which was shown to be optimal when the channel is perfect, is still optimal in the presence of update failures. Our proof relies on tools from Martingale processes, and the construction of a sequence of virtual policies., Comment: To appear in INFOCOM 2018, workshop on Age of Information