Search

A constrained Markov decision process (CMDP) approach is developed for response-adaptive procedures in clinical trials with binary outcomes. The resulting CMDP class of Bayesian response -- adaptive procedures can be used to target a certain objective, e.g., patient benefit or power while using constraints to keep other operating characteristics under control. In the CMDP approach, the constraints can be formulated under different priors, which can induce a certain behaviour of the policy under a given statistical hypothesis, or given that the parameters lie in a specific part of the parameter space. A solution method is developed to find the optimal policy, as well as a more efficient method, based on backward recursion, which often yields a near-optimal solution with an available optimality gap. Three applications are considered, involving type I error and power constraints, constraints on the mean squared error, and a constraint on prior robustness. While the CMDP approach slightly outperforms the constrained randomized dynamic programming (CRDP) procedure known from literature when focussing on type I and II error and mean squared error, showing the general quality of CRDP, CMDP significantly outperforms CRDP when the focus is on type I and II error only., Comment: 35 pages, 18 figures

Sustaining regular and infectious care during an infectious outbreak requires adequate management support for capacity allocation for regular and infectious patients. During the COVID-19 pandemic, hospitals faced severe challenges, including uncertainty concerning the number of infectious patients needing hospitalization and too little regional cooperation. This led to inefficient usage of healthcare capacity. To better prepare for future pandemics, we have developed a decision support system for central regional decision-making on opening and closing (regular care) hospital rooms for infectious patients and assigning new infectious patients to regional hospitals. Since the relabeling of rooms takes some lead time, we make decisions with a stochastic lookahead approach using stochastic programming with sample average approximation based on scenarios of the number of occupied infectious beds and infectious patients needing hospitalization. The lookahead approach produces high-quality decisions by measuring the impact of current decisions on future costs, such as costs for bed shortages, unused beds for infectious patients, and opening and closing rooms. Our simulation study applied to a COVID-19 scenario in the Netherlands, demonstrates that the stochastic lookahead approach outperforms a deterministic approach as well as other heuristic decision rules such as hospitals acting individually and implementing a pandemic unit, i.e., one hospital is designated to take all regional infectious patients until full. Our approach is very flexible and capable of tuning the model parameters to take into account the characteristics of future, yet unknown, pandemics, and supports sustaining regular care by minimizing the strain of infectious care on the available number of beds for regular care., Comment: 27 pages, 3 figures, 11 tables

A sampling-based method is introduced to approximate the Gittins index for a general family of alternative bandit processes. The approximation consists of a truncation of the optimization horizon and support for the immediate rewards, an optimal stopping value approximation, and a stochastic approximation procedure. Finite-time error bounds are given for the three approximations, leading to a procedure to construct a confidence interval for the Gittins index using a finite number of Monte Carlo samples, as well as an epsilon-optimal policy for the Bayesian multi-armed bandit. Proofs are given for almost sure convergence and convergence in distribution for the sampling based Gittins index approximation. In a numerical study, the approximation quality of the proposed method is verified for the Bernoulli bandit and Gaussian bandit with known variance, and the method is shown to significantly outperform Thompson sampling and the Bayesian Upper Confidence Bound algorithms for a novel random effects multi-armed bandit.

This paper considers the cycle maximum in birth-death processes as a stepping stone to characterisation of the cycle maximum in single queues and open Kelly-Whittle networks of queues. For positive recurrent birth-death processes we show that the sequence of sample maxima is stochastically compact. For transient birth-death processes we show that the sequence of sample maxima conditioned on the maximum being finite is stochastically compact. We show that the Markov chain recording the total number of customers in a Kelly-Whittle network is a birth-death process with birth and death rates determined by the normalising constants in a suitably defined sequence of closed networks. Explicit or asymptotic expressions for these normalising constants allow asymptotic evaluation of the birth and death rates, which, in turn, allows characterisation of the cycle maximum in a single busy cycle, and convergence of the sequence of sample maxima for Kelly-Whittle networks of queues., Comment: 10 pages

A Bayesian multivariate model with a structured covariance matrix for multi-way nested data is proposed. This flexible modeling framework allows for positive and for negative associations among clustered observations, and generalizes the well-known dependence structure implied by random effects. A conjugate shifted-inverse gamma prior is proposed for the covariance parameters which ensures that the covariance matrix remains positive definite under posterior analysis. A numerically efficient Gibbs sampling procedure is defined for balanced nested designs, and is validated using two simulation studies. For a top-layer unbalanced nested design, the procedure requires an additional data augmentation step. The proposed data augmentation procedure facilitates sampling latent variables from (truncated) univariate normal distributions, and avoids numerical computation of the inverse of the structured covariance matrix. The Bayesian multivariate (linear transformation) model is applied to two-way nested interval-censored event times to analyze differences in adverse events between three groups of patients, who were randomly allocated to treatment with different stents (BIO-RESORT). The parameters of the structured covariance matrix represent unobserved heterogeneity in treatment effects and are examined to detect differential treatment effects., Comment: 30 pages, 5 figures, 4 tables

We consider the problem of retrieving a reliable estimate of an attribute monitored by a wireless sensor network, where the sensors harvest energy from the environment independently, at random. Each sensor stores the harvested energy in batteries of limited capacity. Moreover, provided they have sufficient energy, the sensors broadcast their measurements in a decentralized fashion. Clients arrive at the sensor network according to a Poisson process and are interested in retrieving a fixed number of sensor measurements, based on which a reliable estimate is computed. We show that the time until an arbitrary sensor broadcasts has a phase-type distribution. Based on this result and the theory of order statistics of phase-type distributions, we determine the probability distribution of the time needed for a client to retrieve a reliable estimate of an attribute monitored by the sensor network. We also provide closed-form expression for the retrieval time of a reliable estimate when the capacity of the sensor battery or the rate at which energy is harvested is asymptotically large. In addition, we analyze numerically the retrieval time of a reliable estimate for various sizes of the sensor network, maximum capacity of the sensor batteries and rate at which energy is harvested. These results show that the energy harvesting rate and the broadcasting rate are the main parameters that influence the retrieval time of a reliable estimate, while deploying sensors with large batteries does not significantly reduce the retrieval time., Comment: 14 pages, 3 figures

We consider a two-node queue modeled as a two-dimensional random walk. In particular, we consider the case that one or both queues have finite buffers. We develop an approximation scheme based on the Markov reward approach to error bounds in order to bound performance measures of such random walks in terms of a perturbed random walk in which the transitions along the boundaries are different from those in the original model and the invariant measure of the perturbed random walk is of product-form. We then apply this approximation scheme to a tandem queue and some variants of this model, for the case that both buffers are finite. We also apply our approximation scheme to a coupled-queue in which only one of the buffers has finite capacity.

We consider homogeneous random walks in the quarter-plane. The necessary conditions which characterize random walks of which the invariant measure is a sum of geometric terms are provided in [2,3]. Based on these results, we first develop an algorithm to check whether the invariant measure of a given random walk is a sum of geometric terms. We also provide the explicit form of the invariant measure if it is a sum of geometric terms. Secondly, for random walks of which the invariant measure is not a sum of geometric terms, we provide an approximation scheme to obtain error bounds for the performance measures. Finally, some numerical examples are provided.

We consider the approximation of the performance of random walks in the quarter-plane. The approximation is in terms of a random walk with a product-form stationary distribution, which is obtained by perturbing the transition probabilities along the boundaries of the state space. A Markov reward approach is used to bound the approximation error. The main contribution of the work is the formulation of a linear program that provides the approximation error.

We consider the invariant measure of homogeneous random walks in the quarter-plane. In particular, we consider measures that can be expressed as an infinite sum of geometric terms. We present necessary conditions for the invariant measure to be a sum of geometric terms. We demonstrate that, under a mild regularity condition, each geometric term must individually satisfy the balance equations in the interior of the state space. We show that the geometric terms in an invariant measure must be the union of finitely many pairwise-coupled sets of infinite cardinality. We further show that for the invariant measure to be a sum of geometric terms, the random walk cannot have transitions to the north, northeast or east. Finally, we show that for an infinite weighted sum of geometric terms to be an invariant measure at least one of the weights must be negative.

A trade-off between two QoS requirements of wireless sensor networks: query waiting time and validity (age) of the data feeding the queries, is investigated. We propose a Continuous Time Markov Decision Process with a drift that trades-off between the two QoS requirements by assigning incoming queries to the wireless sensor network or to the database. To compute an optimal assignment policy, we argue, by means of non-standard uniformization, a discrete time Markov decision process, stochastically equivalent to the initial continuous process. We determine an optimal query assignment policy for the discrete time process by means of dynamic programming. Next, we assess numerically the performance of the optimal policy and show that it outperforms in terms of average assignment costs three other heuristics, commonly used in practice. Lastly, the optimality of the our model is confirmed also in the case of real query traffic, where our proposed policy achieves significant cost savings compared to the heuristics., Comment: 27 pages, 20 figures

We consider the invariant measure of homogeneous random walks in the quarter-plane. In particular, we consider measures that can be expressed as a finite linear combination of geometric terms and present conditions on the structure of these linear combinations such that the resulting measure may yield an invariant measure of a random walk. We demonstrate that each geometric term must individually satisfy the balance equations in the interior of the state space and further show that the geometric terms in an invariant measure must have a pairwise-coupled structure. Finally, we show that at least one of the coefficients in the linear combination must be negative.

We consider the invariant measure of a homogeneous continuous- time Markov process in the quarter-plane. The basic solutions of the global balance equation are the geometric distributions. We first show that the invariant measure can not be a finite linear combination of basic geometric distributions, unless it consists of a single basic geo- metric distribution. Second, we show that a countable linear combina- tion of geometric terms can be an invariant measure only if it consists of pairwise-coupled terms. As a consequence, we obtain a complete characterization of all countable linear combinations of geometric dis- tributions that may yield an invariant measure for a homogeneous continuous-time Markov process in the quarter-plane., Comment: The author has submitted a better version of this paper. However, the author did not know the new version should replace the old version. Therefore, there is another paper in arXiv with the same topic

In this paper, we will develop a tool to analyze polling systems with the autonomous-server, the time-limited, and the k-limited service discipline. It is known that these disciplines do not satisfy the well-known branching property in polling system, therefore, hardly any exact result exists in the literature for them. Our strategy is to apply an iterative scheme that is based on relating in closed-form the joint queue-length at the beginning and the end of a server visit to a queue. These kernel relations are derived using the theory of absorbing Markov chains. Finally, we will show that our tool works also in the case of a tandem queueing network with a single server that can serve one queue at a time.