Search

The early sections of this paper present an analysis of a Markov decision model that is known as the multi-armed bandit under the assumption that the utility function of the decision maker is either linear or exponential. The analysis includes efficient procedures for computing the expected utility associated with the use of a priority policy and for identifying a priority policy that is optimal. The methodology in these sections is novel, building on the use of elementary row operations. In the later sections of this paper, the analysis is adapted to accommodate constraints that link the bandits.

For an upper semi-continuous set-valued mapping from one topological space to another and for a lower semi-continuous function defined on the product of these spaces, Berge's theorem states lower semi-continuity of the minimum of this function taken over the image sets. It assumes that the image sets are compact. For Hausdorff topological spaces, this paper extends Berge's theorem to set-valued mappings with possible noncompact image sets and studies relevant properties of minima.

This paper presents sufficient conditions for the existence of stationary optimal policies for average-cost Markov Decision Processes with Borel state and action sets and with weakly continuous transition probabilities. The one-step cost functions may be unbounded, and action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount-optimal and average-cost optimal policies and descriptions of properties of value functions and sets of optimal actions, (ii) a sufficient condition for the average-cost optimality of a stationary policy in the form of optimality inequalities, and (iii) approximations of average-cost optimal actions by discount-optimal actions., Comment: 26 pages

Consider a measurable space with a finite vector measure. This measure defines a mapping of the $\sigma$-field into a Euclidean space. According to Lyapunov's convexity theorem, the range of this mapping is compact and, if the measure is atomless, this range is convex. Similar ranges are also defined for measurable subsets of the space. We show that the union of the ranges of all subsets having the same given vector measure is also compact and, if the measure is atomless, it is convex. We further provide a geometrically constructed convex compactum in the Euclidean space that contains this union. The equality of these two sets, that holds for two-dimensional measures, can be violated in higher dimensions., Comment: 7 pages, 1 figure

Consider a measurable space with an atomless finite vector measure. This measure defines a mapping of the $\sigma$-field into an Euclidean space. According to the Lyapunov convexity theorem, the range of this mapping is a convex compactum. Similar ranges are also defined for measurable subsets of the space. Two subsets with the same vector measure may have different ranges. We investigate the question whether, among all the subsets having the same given vector measure, there always exists a set with the maximal range of the vector measure. The answer to this question is positive for two-dimensional vector measures and negative for higher dimensions. We use the existence of maximal ranges to strengthen the Dvoretzky-Wald-Wolfowitz purification theorem for the case of two measures., Comment: 15 pages, 1 figure; minor revision, references added, typos corrected

We have presented an optimal buffer sizing and buffer insertion methodology which uses stochastic models of the architecture and Continuous Time Markov Decision Processes CTMDPs. Such a methodology is useful in managing the scarce buffer resources available on chip as compared to network based data communication which can have large buffer space. The modeling of this problem in terms of a CT-MDP framework lead to a nonlinear formulation due to usage of bridges in the bus architecture. We present a methodology to split the problem into several smaller though linear systems and we then solve these subsystems., Comment: Submitted on behalf of EDAA (http://www.edaa.com/)