Search

Fairness plays a crucial role in various multi-agent systems (e.g., communication networks, financial markets, etc.). Many multi-agent dynamical interactions can be cast as Markov Decision Processes (MDPs). While existing research has focused on studying fairness in known environments, the exploration of fairness in such systems for unknown environments remains open. In this paper, we propose a Reinforcement Learning (RL) approach to achieve fairness in multi-agent finite-horizon episodic MDPs. Instead of maximizing the sum of individual agents' value functions, we introduce a fairness function that ensures equitable rewards across agents. Since the classical Bellman's equation does not hold when the sum of individual value functions is not maximized, we cannot use traditional approaches. Instead, in order to explore, we maintain a confidence bound of the unknown environment and then propose an online convex optimization based approach to obtain a policy constrained to this confidence region. We show that such an approach achieves sub-linear regret in terms of the number of episodes. Additionally, we provide a probably approximately correct (PAC) guarantee based on the obtained regret bound. We also propose an offline RL algorithm and bound the optimality gap with respect to the optimal fair solution. To mitigate computational complexity, we introduce a policy-gradient type method for the fair objective. Simulation experiments also demonstrate the efficacy of our approach.

We theoretically explore the Floquet generation of Majorana end modes~(MEMs) (both regular $0$- and anomalous $\pi$-modes) implementing a periodic sinusoidal modulation in chemical potential in an experimentally feasible setup based on one-dimensional chain of magnetic impurity atoms having spin spiral configuration fabricated on the surface of most common bulk $s$-wave superconductor. We obtain a rich phase diagram in the parameter space, highlighting the possibility of generating multiple $0$-/$\pi$-MEMs localized at the end of the chain. We also study the real-time evolution of these emergent MEMs, especially when they start to appear in the time domain. These MEMs are topologically characterized by employing the dynamical winding number. We also discuss the possible experimental parameters in connection to our model. Our work paves the way to realize the Floquet MEMs in a magnet-superconductor heterostructure., Comment: 7.5 Pages + 5 PDF figures (Main Text), 4 Pages + 3 PDF figures (Supplementary Material), Comments are welcome

We report on a realistic and rather general scheme where noncollinear magnetic textures proximitized with the most common $s$-wave superconductor can appear as the alternative to $p$-wave superconductor{--}the prime proposal to realize two-dimensional (2D) Kitaev model for topological superconductors (TSCs) hosting Majorana flat edge mode (MFEM). A general minimal Hamiltonian suitable for magnet/superconductor heterostructures reveals robust MFEM within the gap of Shiba bands, spatially localized at the edges of a 2D magnetic domain of spin-spiral. We finally verify this concept by considering Mn (Cr) monolayer grown on a $s$-wave superconducting substrate, Nb(110) under strain (Nb(001)). In both 2D cases, the antiferromagnetic spin-spiral solutions exhibit robust MFEM at certain domain edges. This approach, particularly when the MFEM appears in the TSC phase for such heterostructure materials, offers a perspective to extend the realm of the TSC in 2D., 7.5 Pages + 5 PDF figures (Main Text), 11 Pages + 8 PDF figures (Supplementary Material), Comments are welcome

Autonomous vehicles will be an integral part of ride-sharing services in the future. This setting differs from traditional ride-sharing marketplaces because of the absence of the supply side (drivers). However, it has far-reaching consequences because in addition to pricing, players now have to make decisions on how to distribute fleets across network locations and re-balance vehicles in order to serve future demand. In this paper, we explore a duopoly setting in the ride-sharing marketplace where the players have fully autonomous fleets. Each ride-service provider (RSP)'s prices depend on the prices and the supply of the other player. We formulate their decision-making problems using a game-theoretic setup where each player seeks to find the optimal prices and supplies at each node while considering the decisions of the other player. This leads to a scenario where the players' optimization problems are coupled and it is challenging to find the equilibrium. We characterize the types of demand functions (e.g.: linear) for which this game admits an exact potential function and can be solved efficiently. For other types of demand functions, we propose an iterative algorithm to compute the equilibrium. We conclude by providing numerical insights into how different kinds of equilibria would play out in the market when the players are asymmetric. Our numerical evaluations also provide insights into how the regulator needs to consider network effects while deciding regulation in order to avoid unfavorable outcomes., Submitted to MobiHoc'23

We study model-free reinforcement learning (RL) algorithms in episodic non-stationary constrained Markov Decision Processes (CMDPs), in which an agent aims to maximize the expected cumulative reward subject to a cumulative constraint on the expected utility (cost). In the non-stationary environment, reward, utility functions, and transition kernels can vary arbitrarily over time as long as the cumulative variations do not exceed certain variation budgets. We propose the first model-free, simulator-free RL algorithms with sublinear regret and zero constraint violation for non-stationary CMDPs in both tabular and linear function approximation settings with provable performance guarantees. Our results on regret bound and constraint violation for the tabular case match the corresponding best results for stationary CMDPs when the total budget is known. Additionally, we present a general framework for addressing the well-known challenges associated with analyzing non-stationary CMDPs, without requiring prior knowledge of the variation budget. We apply the approach for both tabular and linear approximation settings.

We consider a multi-agent episodic MDP setup where an agent (leader) takes action at each step of the episode followed by another agent (follower). The state evolution and rewards depend on the joint action pair of the leader and the follower. Such type of interactions can find applications in many domains such as smart grids, mechanism design, security, and policymaking. We are interested in how to learn policies for both the players with provable performance guarantee under a bandit feedback setting. We focus on a setup where both the leader and followers are {\em non-myopic}, i.e., they both seek to maximize their rewards over the entire episode and consider a linear MDP which can model continuous state-space which is very common in many RL applications. We propose a {\em model-free} RL algorithm and show that $\tilde{\mathcal{O}}(\sqrt{d^3H^3T})$ regret bounds can be achieved for both the leader and the follower, where $d$ is the dimension of the feature mapping, $H$ is the length of the episode, and $T$ is the total number of steps under the bandit feedback information setup. Thus, our result holds even when the number of states becomes infinite. The algorithm relies on {\em novel} adaptation of the LSVI-UCB algorithm. Specifically, we replace the standard greedy policy (as the best response) with the soft-max policy for both the leader and the follower. This turns out to be key in establishing uniform concentration bound for the value functions. To the best of our knowledge, this is the first sub-linear regret bound guarantee for the Markov games with non-myopic followers with function approximation., arXiv admin note: text overlap with arXiv:2206.11889

Traffic congestion in urban environment is one of the most critical issue for drivers and city planners for both environment and efficiency reasons. Traffic lights are one of the main tools used to regulate traffic by diverting the drivers between different paths. Rational drivers, in turn, react to the traffic light duration by evaluating their options and, if necessary, by changing direction in order to reach their destination quicker. In this paper, we introduce a macroscopic traffic model for urban intersections that incorporates this rational behavior of the drivers. Then, we exploit it to show that, by providing additional information about the expected red-time duration to the drivers, one can decrease the amount of congestion in the network and the overall length of the queues at the intersections. Additionally, we develop a control policy for the traffic lights that exploits the reaction of the drivers in order to divert them to a different route to further increase the performances. These claims are supported by extensive numerical simulations.

To fully utilize the abundant spectrum resources in millimeter wave (mmWave), Beam Alignment (BA) is necessary for large antenna arrays to achieve large array gains. In practical dynamic wireless environments, channel modeling is challenging due to time-varying and multipath effects. In this paper, we formulate the beam alignment problem as a non-stationary online learning problem with the objective to maximize the received signal strength under interference constraint. In particular, we employ the non-stationary kernelized bandit to leverage the correlation among beams and model the complex beamforming and multipath channel functions. Furthermore, to mitigate interference to other user equipment, we leverage the primal-dual method to design a constrained UCB-type kernelized bandit algorithm. Our theoretical analysis indicates that the proposed algorithm can adaptively adjust the beam in time-varying environments, such that both the cumulative regret of the received signal and constraint violations have sublinear bounds with respect to time. This result is of independent interest for applications such as adaptive pricing and news ranking. In addition, the algorithm assumes the channel is a black-box function and does not require any prior knowledge for dynamic channel modeling, and thus is applicable in a variety of scenarios. We further show that if the information about the channel variation is known, the algorithm will have better theoretical guarantees and performance. Finally, we conduct simulations to highlight the effectiveness of the proposed algorithm.

We propose an event-triggered framework for deciding the traffic light at each lane in a mixed autonomy scenario. We deploy the decision after a suitable delay, and events are triggered based on the satisfaction of a predefined set of conditions. We design the trigger conditions and the delay to increase the vehicles’ throughput. This way, we achieve full exploitation of autonomous vehicles (AVs) potential. The ultimate goal is to obtain vehicle-flows led by AVs at the head. We formulate the decision process of the traffic intersection controller as a deterministic delayed Markov decision process, i.e., the action implementation and evaluation are delayed. We propose a Reinforcement Learning based model-free algorithm to obtain the optimal policy. We show - by simulations - that our algorithm converges, and significantly reduces the average wait-time and the queues length as the fraction of the AVs increases. Our algorithm outperforms our previous work [1] by a quite significant amount.

We study the constrained reinforcement learning problem, in which an agent aims to maximize the expected cumulative reward subject to a constraint on the expected total value of a utility function. In contrast to existing model-based approaches or model-free methods accompanied with a `simulator', we aim to develop the first model-free, simulator-free algorithm that achieves a sublinear regret and a sublinear constraint violation even in large-scale systems. To this end, we consider the episodic constrained Markov decision processes with linear function approximation, where the transition dynamics and the reward function can be represented as a linear function of some known feature mapping. We show that $\tilde{\mathcal{O}}(\sqrt{d^3H^3T})$ regret and $\tilde{\mathcal{O}}(\sqrt{d^3H^3T})$ constraint violation bounds can be achieved, where $d$ is the dimension of the feature mapping, $H$ is the length of the episode, and $T$ is the total number of steps. Our bounds are attained without explicitly estimating the unknown transition model or requiring a simulator, and they depend on the state space only through the dimension of the feature mapping. Hence our bounds hold even when the number of states goes to infinity. Our main results are achieved via novel adaptations of the standard LSVI-UCB algorithms. In particular, we first introduce primal-dual optimization into the LSVI-UCB algorithm to balance between regret and constraint violation. More importantly, we replace the standard greedy selection with respect to the state-action function in LSVI-UCB with a soft-max policy. This turns out to be key in establishing uniform concentration for the constrained case via its approximation-smoothness trade-off. We also show that one can achieve an even zero constraint violation while still maintaining the same order with respect to $T$., Comment: Accepted and Published at the 36th Neural Information Processing Systems (NeurIPS'22). Section J (where different episodes may start from different states) is added in this version

We consider an optimal traffic-light control framework for urban traffic intersections to alleviate congestion phenomena. We analyze a scenario in which we provide drivers with information about the waiting time at the intersections. We model the drivers’ lane-changing information-based behavior as the solution of a convex optimization problem. We compute the optimal traffic-light control mechanism as the solution to a bi-level optimization problem. We provide a complete analysis in terms of (i) the existence of a solution; (ii) an iterative algorithm to compute it; (iii) sufficient conditions for the solution’s uniqueness and the algorithm’s convergence. Early simulation results show the proposed control scheme’s effectiveness compared with an optimal control algorithm in the absence of waiting-time information.

We consider an intersection zone where autonomous vehicles (AVs) and human-driven vehicles (HDVs) can be present. As a new vehicle arrives, the traffic controller needs to decide and impose an optimal sequence of the vehicles that will exit the intersection zone. The traffic controller can send information regarding the time at which an AV can cross the intersection; however, the traffic controller can not communicate with the HDVs, rather the HDVs can only be controlled using the traffic lights. We formulate the problem as an integer constrained non-linear optimization problem where the traffic-intersection controller only communicates with a subset of the AVs. Since the number of possible combinations increases exponentially with the number of vehicles in the system, we relax the original problem and proposes an algorithm that gives the optimal solution of the relaxed problem and yet only scales linearly with the number of vehicles in the system. The numerical evaluation shows that our algorithm outperforms the First-In-First-Out (FIFO) algorithm.

Autonomous vehicles (AVs) are attractive for ride service providers (RSPs) in part because they eliminate the need to compete for human drivers. We investigate a scenario where two RSPs with AVs compete for customers. We model the problem as a game where the RSPs select prices for each origin-destination pair over multiple time periods in an underlying graph representing the customers' desired trips. Each RSP also decides the number of AVs to be stationed at each node at each time period to serve the customers' demands. The number of customers who avail of the service of an RSP depends on the price selected by the RSP and its competitor. Since the strategy choices available to an RSP depends on its competitor, we seek to compute a Generalized Nash equilibrium (GNE). We show that there may be multiple GNEs. However, when an RSP selects prices in order to deter its competitor when it is not serving a source-destination pair, the game has a potential function and admits a unique GNE. We also compare the competitive prices with a monopoly price where only one RSP is in the market. Numerically, we show that if a network consists of two equal-size spatial clusters of demand where the demand between clusters is low, the RSPs may partition the market, i.e, one cluster is served by only one RSP. Hence, the competitive price may become close to the monopoly price., Submitted in IEEE Conference on Decision and Control

Users can now give back energies to the grid using distributed resources. Proper incentive mechanisms are required for such users, also known as prosumers, in order to maximize the sell-back amount while maintaining the retailer's profit. However, all the existing literature considers expected utility theory (EUT) where they assume that prosumers maximize their expected payoff. We consider prospect theory (PT) which models the behavior of humans in the face of uncertainty in a better manner. We show that in a day-ahead contract pricing mechanism, the actual optimal value of contract and the sell-back amount may be smaller compared to the one computed by the EUT. We also propose a lottery-based mechanism and show that such a mechanism can increase the sell-back amount while increasing the retailer's savings compared to day-ahead contract pricing., Accepted in ECC'21

We consider a scenario where a retailer can set different prices for different consumers in a smart grid. The retailer's objective is to maximize the revenue, minimize the operating cost, and maximize the consumer's welfare. The retailer wants to optimize a convex combination of the above objectives using price signals specific to each consumer. However, variability in unit prices across consumers is bounded by a parameter $\eta$, hence limiting the discrimination. We formulate the pricing problem as a Stackelberg game where the retailer is the leader and consumers are followers. Since the retailer's optimization problem turns out to be non-convex, we convexify it via relaxations. We provide performance guarantees for the relaxations in the asymptotic sense (when number of consumers tends to $\infty$). Further, we show that despite the variability in pricing, the pricing scheme proposed by our model is fair as higher prices are charged to consumers who have higher willingness for demand. We extend our analysis to the scenario where consumers can feed energy back to the grid via net-metering. We show that our pricing policy promotes fairness even in this scenario as prosumers who contribute more to the grid, are given large cuts on buying rates. The policy is also found to incentivize more prosumers to invest in renewable energy, thus encouraging sustainability., Comment: Under review

Stochastic games provide a framework for interactions among multiple agents and enable a myriad of applications. In these games, agents decide on actions simultaneously, the state of every agent moves to the next state, and each agent receives a reward. However, finding an equilibrium (if exists) in this game is often difficult when the number of agents becomes large. This paper focuses on finding a mean-field equilibrium (MFE) in an action coupled stochastic game setting in an episodic framework. It is assumed that the impact of the other agents' can be assumed by the empirical distribution of the mean of the actions. All agents know the action distribution and employ lower-myopic best response dynamics to choose the optimal oblivious strategy. This paper proposes a posterior sampling based approach for reinforcement learning in the mean-field game, where each agent samples a transition probability from the previous transitions. We show that the policy and action distributions converge to the optimal oblivious strategy and the limiting distribution, respectively, which constitute an MFE.

We consider users which may have renewable energy harvesting devices, or distributed generators. Such users can behave as consumer or producer (hence, we denote them as prosumers) at different time instances. A prosumer may sell the energy to other prosumers in exchange of money. We consider a demand response model, where the price of conventional energy depends on the total demand of all the prosumers at a certain time. A prosumer depending on its own utility has to select the amount of energy it wants to buy either from the grid or from other prosumers, or the amount of excess energy it wants to sell to other prosumers. However, the strategy, and the payoff of a prosumer inherently depends on the strategy of other prosumers as a prosumer can only buy if the other prosumers are willing to sell. We formulate the problem as a coupled constrained game, and seek to obtain the generalized Nash equilibrium. We show that the game is a concave potential game and show that there exists a unique generalized Nash equilibrium. We consider that a platform will set the price for distributed interchange of energy among the prosumers in order to minimize the consumption of the conventional energy. We propose a distributed algorithm where the platform sets a price to each prosumer, and then each prosumer at a certain time only optimizes its own payoff. The prosumer then updates the price depending on the supply and demand for each prosumer. We show that the algorithm converges to an optimal generalized Nash equilibrium. The distributed algorithm also provides an optimal price for the exchange market., Comment: 9 pages, 4 figures

We propose a robust scheme for streaming 360-degree immersive videos to maximize the quality of experience (QoE). Our streaming approach introduces a holistic analytical framework built upon the formal method of stochastic optimization. We propose a robust algorithm which provides a streaming rate such that the video quality degrades below that rate with very low probability even in presence of uncertain head movement, and bandwidth. It assumes the knowledge of the viewing probability of different portions (tiles) of a panoramic scene. Such probabilities can be easily derived from crowdsourced measurements performed by 360 video content providers. We then propose efficient methods to solve the problem at runtime while achieving a bounded optimality gap (in terms of the QoE). We implemented our proposed approaches using emulation. Using real users' head movement traces and real cellular bandwidth traces, we show that our algorithms significantly outperform the baseline algorithms by at least in $30\%$ in the QoE metric. Our algorithm gives a streaming rate which is $50\%$ higher compared to the baseline algorithms when the prediction error is high., Comment: arXiv admin note: text overlap with arXiv:1704.08215

In the 360-degree immersive video, a user only views a part of the entire raw video frame based on her viewing direction. However, today's 360-degree video players always fetch the entire panoramic view regardless of users' head movement, leading to significant bandwidth waste that can be potentially avoided. In this paper, we propose a novel adaptive streaming scheme for 360-degree videos. The basic idea is to fetch the invisible portion of a video at the lowest quality based on users' head movement prediction and to adaptively decide the video playback quality for the visible portion based on bandwidth prediction. Doing both in a robust manner requires overcome a series of challenges, such as jointly considering the spatial and temporal domains, tolerating prediction errors, and achieving low complexity. To overcome these challenges, we first define quality of experience (QoE) metrics for adaptive 360-degree video streaming. We then formulate an optimization problem and solve it at a low complexity. The algorithm strategically leverages both future bandwidth and the distribution of users' head positions to determine the quality level of each tile (i.e., a sub-area of a raw frame). We further provide theoretical proof showing that our algorithm achieves optimality under practical assumptions. Numerical results show that our proposed algorithms significantly boost the user QoE by at least 20\% compared to baseline algorithms.

In a secondary spectrum market primaries set prices for their unused channels to the secondaries. The payoff of a primary depends on the availability of unused channels of its competitors. We consider a model were a primary can acquire its competitor's channel state information (C-CSI) at a cost. We formulate a game between two primaries where each primary decides whether to acquire C-CSI or not and then selects its price based on that. We first characterize the Nash Equilibrium (NE) of this game for a symmetric model where the C-CSI is perfect. We show that the payoff of a primary is independent of the C-CSI acquisition cost. We then generalize our analysis to allow for imperfect estimation and cases where the two primaries have different C-CSI costs or different channel availabilities. Our results show interestingly that the payoff of a primary increases when there is estimation error. We also show that surprisingly, the expected payoff of a primary may decrease when the C-CSI acquisition cost decreases when primaries have different availabilities., Shorter version will be presented in ISIT'16. Submitted to JSAC Special Issue on Spectrum Sharing, 2016. Impacts of Estimation errors, different C-CSI acquisition costs and different availabilities are now considered

The economic model of the Internet of Things (IoT) consists of end users, advertisers and three different kinds of providers--IoT service provider (IoTSP), Wireless service provider (WSP) and cloud service provider (CSP). We investigate three different kinds of interactions among the providers. First, we consider that the IoTSP prices a bundled service to the end-users, and the WSP and CSP pay the IoTSP (push model). Next, we consider the model where the end-users independently pay the each provider (pull model). Finally, we consider a hybrid model of the above two where the IoTSP and WSP quote their prices to the end-users, but the CSP quotes its price to the IoTSP. We characterize and quantify the impact of the advertisement revenue on the equilibrium pricing strategy and payoff of providers, and corresponding demands of end users in each of the above interaction models. Our analysis reveals that the demand of end-users, and the payoffs of the providers are non decreasing functions of the advertisement revenue. For sufficiently high advertisement revenue, the IoTSP will offer its service free of cost in each interaction model. However, the payoffs of the providers, and the demand of end-users vary across different interaction models. Our analysis shows that the demand of end-users, and the payoff of the WSP are the highest in the pull (push, resp.) model in the low (high, resp.) advertisement revenue regime. The payoff of the IoTSP is always higher in the pull model irrespective of the advertisement revenue. The payoff of the CSP is the highest in the hybrid model in the low advertisement revenue regime. However, in the high advertisement revenue regime the payoff of the CSP in the hybrid model or in the push model can be higher depending on the equilibrium chosen in the push model., Submitted in IEEE Transactions on Automatic Control

International audience; We study a mean field approximation of the M/M/$\infty$ queuing system. This queue is often used to model the number of cellular phone users in a cell. We assume that congestion here has a positive impact on the performance so that the more there are users, the less it is costly to offer a service per cell phone, for example, if a base station broadcasts a film then the cost per customer decreases. We obtain closed-form formulas for the equilibria. We show that the mean-field approximation becomes tight as the workload increases, thus the results obtained for the mean-field model well approximate the discrete one.

We investigate a spectrum oligopoly market where each primary seeks to sell secondary access to its channel at multiple locations. Transmission qualities of a channel evolve randomly. Each primary needs to select a price and a set of non-interfering locations (which is an independent set in the conflict graph of the region) at which to offer its channel without knowing the transmission qualities of the channels of its competitors. We formulate the above problem as a non-cooperative game. We consider two scenarios-i) when the region is small, ii) when the region is large. In the first setting, we focus on a class of conflict graphs, known as mean valid graphs which commonly arise when the region is small. We explicitly compute a symmetric Nash equilibrium (NE); the NE is threshold type in that primaries only choose independent set whose cardinality is greater than a certain threshold. The threshold on the cardinality increases with increase in quality of the channel on sale. We show that the symmetric NE strategy profile is unique in a special class of conflict graphs (linear graph). In the second setting, we consider node symmetric conflict graphs which arises when the number of locations is large (potentially, infinite). We explicitly compute a symmetric NE that randomizes equally among the maximum independent sets at a given channel state vector. In the NE a primary only selects the maximum independent set at a given channel state vector. We show that the two symmetric NEs computed in two settings exhibit important structural difference., Added several results from the other versions. Submitted to IEEE Transactions on Information Theory. 63 pages one column, 22 figures

We investigate a spectrum oligopoly market where primaries lease their channels to secondaries in lieu of financial remuneration. Transmission quality of a channel evolves randomly. Each primary has to select the price it would quote without knowing the transmission qualities of its competitors' channels. Each secondary buys a channel depending on the price and the transmission quality a channel offers. We formulate the price selection problem as a non co-operative game with primaries as players. In the one-shot game, we show that there exists a unique symmetric Nash Equilibrium(NE) strategy profile and explicitly compute it. Our analysis reveals that under the NE strategy profile a primary prices its channel to render high quality channel more preferable to the secondary; this negates the popular belief that prices ought to be selected to render channels equally preferable to the secondary regardless of their qualities. We show the loss of revenue in the asymptotic limit due to the non co-operation of primaries. In the repeated version of the game, we characterize a subgame perfect NE where a primary can attain a payoff arbitrarily close to the payoff it would obtain when primaries co-operate., Comment: Accepted for publication in IEEE/ACM Transactions on Networking. 41 pages single column format.Conference version is available at arXiv:1305.3351

We propose an event-triggered framework for deciding the traffic light at each lane in a mixed autonomy scenario. We deploy the decision after a suitable delay, and events are triggered based on the satisfaction of a predefined set of conditions. We design the trigger conditions and the delay to increase the vehicles’ throughput. This way, we achieve full exploitation of autonomous vehicles (AVs) potential. The ultimate goal is to obtain vehicle-flows led by AVs at the head. We formulate the decision process of the traffic intersection controller as a deterministic delayed Markov decision process, i.e., the action implementation and evaluation are delayed. We propose a Reinforcement Learning based model-free algorithm to obtain the optimal policy. We show - by simulations - that our algorithm converges, and significantly reduces the average wait-time and the queues length as the fraction of the AVs increases. Our algorithm outperforms our previous work [1] by a quite significant amount.

We consider a mixed autonomy scenario where the traffic intersection controller decides whether the traffic light will be green or red at each lane for multiple traffic-light blocks. The objective of the traffic intersection controller is to minimize the queue length at each lane and maximize the outflow of vehicles over each block. We consider that the traffic intersection controller informs the autonomous vehicle (AV) whether the traffic light will be green or red for the future traffic-light block. Thus, the AV can adapt its dynamics by solving an optimal control problem. We model the decision process of the traffic intersection controller as a deterministic delay Markov decision process owing to the delayed action by the traffic controller. We propose Reinforcement-learning based algorithm to obtain the optimal policy. We show - empirically - that our algorithm converges and reduces the energy costs of AVs drastically as the traffic controller communicates with the AVs., Accepted for Publication at 24th IEEE International Conference on Intelligent Transportation (ITSC'2021)