Search

A solution approach is proposed for the interday problem of assigning chemotherapy sessions at a network of treatment centres with the goal of increasing the cost-efficiency of system-wide capacity use. This network-based scheduling procedure is subject to the condition that both the first and last sessions of a patient's treatment protocol are administered at the same centre the patient is referred to by their oncologist. All intermediate sessions may be administered at other centres. It provides a systematic way of identifying effective multi-appointment scheduling policies that exploit the total capacity of a networked system, allowing patients to be treated at centres other than their home centre. The problem is modelled as a Markov decision process which is then solved approximately using techniques of approximate dynamic programming. The benefits of the approach are evaluated and compared through simulation with the existing manual scheduling procedures at two treatment centres in Santiago, Chile. The results suggest that the approach would obtain a 20% reduction in operating costs for the whole system and cut existing first-session waiting times by half. A key conclusion, however, is that a network-based scheduling procedure brings no real benefits if it is not implemented in conjunction with a proactive assignment policy like the one proposed in this paper. [ABSTRACT FROM AUTHOR]

This paper extends the results provided in Jasso-Fuentes et al. (Appl Math Optim 81(2):409–441, 2020b) and Jasso-Fuentes et al. (Pure Appl Funct Anal 9(3):675–704, 2024) regarding the study of discrete-time hybrid stochastic models with general spaces and total discounted payoffs. This extension incorporates the handling of negative and/or unbounded costs per stage. In particular, it encompasses interesting applications, such as scenarios where the controller optimizes net costs, social welfare costs, or distances between points. These situations arise when assumptions of both non-negativeness and boundedness on the cost per stage do not apply. Our proposal relies on Lyapunov-like conditions, enabling, among other aspects, the finiteness of the value function and the existence of solutions to the associated dynamic programming equation. This equation is crucial for deriving optimal control policies. To illustrate our theory, we include an example in inventory-manufacturing management, highlighting its evident hybrid nature. [ABSTRACT FROM AUTHOR]

Markov chains and Markov decision processes have been widely used to model the behavior of computer systems with probabilistic aspects. Numerical and iterative methods are commonly used to analyze these models. Many efforts have been made in recent decades to improve the efficiency of these numerical methods. In this paper, focusing on Markov models with non-sparse structure, a new set of heuristics is proposed for prioritizing model states with the aim of reducing the total computation time. In these heuristics, a set of simulation runs are used for statistical analysis of the effect of each state on the required values of the other states. Under this criterion, the priority of each state in updating its values is determined. The proposed heuristics provide a state ordering that improves the value propagation among the states. The proposed methods are also extended for very large models where disk-based techniques are required to analyze the models. Experimental results show that our proposed methods in this paper reduce the running times of the iterative methods for most cases of non-sparse models. [ABSTRACT FROM AUTHOR]

We present novel techniques for simultaneous task allocation and planning in multi-robot systems operating under uncertainty. By performing task allocation and planning simultaneously, allocations are informed by individual robot behaviour, creating more efficient team behaviour. We go beyond existing work by planning for task reallocation across the team given a model of partial task satisfaction under potential robot failures and uncertain action outcomes. We model the problem using Markov decision processes, with tasks encoded in co-safe linear temporal logic, and optimise for the expected number of tasks completed by the team. To avoid the inherent complexity of joint models, we propose an alternative model that simultaneously considers task allocation and planning, but in a sequential fashion. We then build a joint policy from the sequential policy obtained from our model, thus allowing for concurrent policy execution. Furthermore, to enable adaptation in the case of robot failures, we consider replanning from failure states and propose an approach to preemptively replan in an anytime fashion, replanning for more probable failure states first. Our method also allows us to quantify the performance of the team by providing an analysis of properties, such as the expected number of completed tasks under concurrent policy execution. We implement and extensively evaluate our approach on a range of scenarios. We compare its performance to a state-of-the-art baseline in decoupled task allocation and planning: sequential single-item auctions. Our approach outperforms the baseline in terms of computation time and the number of times replanning is required on robot failure. [ABSTRACT FROM AUTHOR]

Nowadays, logistics service providers (LSPs) increasingly consider using a crowdsourced workforce on the last mile to fulfill customers' expectations regarding same-day or on-demand delivery at reduced costs. The crowdsourced workforce's availability is, however, uncertain. Therefore, LSPs often hire additional fixed employees to perform deliveries when the availability of crowdsourced drivers is low. In this context, the reliability versus flexibility trade-off which LSPs face over a longer period, for example, a year, remains unstudied. Against this background, we jointly study a workforce planning problem that considers salaried drivers (SDs) and the temporal development of the crowdsourced driver (CD) fleet over a long-term time horizon. We consider two types of CDs, dedicated gig-drivers (DDs) and opportunistic gig-drivers (ODs). While DDs are not sensitive to the request's destination and typically exhibit high availability, ODs only serve requests whose origin and destination coincide with their own private route's origin and destination. Moreover, to account for time horizon-specific dynamics, we consider stochastic turnover for both SDs and CDs as well as stochastic CD fleet growth. We formulate the resulting workforce planning problem as a Markov decision process whose reward function reflects total costs, that is, wages and operational costs arising from serving demand with SDs and CDs, and solve it via approximate dynamic programming. Applying our approach to an environment based on real-world demand data from GrubHub, we find that in fleets consisting of SDs and CDs, approximate dynamic programming (ADP)-based hiring policies can outperform myopic hiring policies by up to 19 % and lookahead policies with perfect knowledge of future information by up to 10 % in total costs. In the studied setting, we observed that DDs reduce the LSP's total costs more than ODs. When we account for CDs' increased resignation probability when not being matched with enough requests, the amount of required SDs increases. [ABSTRACT FROM AUTHOR]

Modeling as real options the operations of energy production companies that operate in wholesale markets gives rise to a challenging Markov decision process. In "Least Squares Monte Carlo and Pathwise Optimization for Merchant Energy Production," Yang, Nadarajah, and Secomandi study the performance of two reinforcement learning techniques that can be used to determine feasible operating policies and optimality bounds for this model, namely least squares Monte Carlo and pathwise optimization, extending the applicability of the latter method beyond optimal stopping by using principal component analysis and block coordinate descent. They find that both approaches lead to near optimal policies, but pathwise optimization outperforms least squares Monte Carlo in terms of dual bounds at the expense of more sizable computational requirements. These findings have potential relevance for managers of energy production assets that use analytics to optimize their operations and researchers interested in broadening the scope of pathwise optimization. We study merchant energy production modeled as a compound switching and timing option. The resulting Markov decision process is intractable. Least squares Monte Carlo combined with information relaxation and duality is a state-of-the-art reinforcement learning methodology to obtain operating policies and optimality gaps for related models. Pathwise optimization is a competing technique developed for optimal stopping settings, in which it typically provides superior results compared with this approach, albeit with a larger computational effort. We apply these procedures to merchant energy production. Using pathwise optimization requires methodological extensions. We use principal component analysis and block coordinate descent in novel ways to respectively precondition and solve the ensuing ill-conditioned and large-scale linear program, which even a cutting-edge commercial solver is unable to handle directly. Both techniques yield near optimal operating policies on realistic ethanol production instances. However, at the cost of both considerably longer run times and greater memory usage, which limits the number of stages of the instances that it can handle, pathwise optimization leads to substantially tighter dual bounds compared with least squares Monte Carlo, even when specified in a simple fashion, complementing it in this case. Thus, it plays a critical role in obtaining small optimality gaps. Our numerical observations on the magnitudes of these bound improvements differ from what is currently known. This research has potential relevance for other commodity merchant operations contexts and motivates additional algorithmic work in the area of pathwise optimization. Funding: This work was supported by the Division of Civil, Mechanical, and Manufacturing Innovation, National Science Foundation [Grant 1761742] and a grant from the Scott Institute for Energy Innovation at Carnegie Mellon University. S. Nadarajah acknowledges support from the University of Illinois at Chicago College of Business [Dean's Research Grant]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/opre.2018.0341. [ABSTRACT FROM AUTHOR]

We provide a framework to analyze control policies for the restless Markovian bandit model under both finite and infinite time horizons. We show that when the population of arms goes to infinity, the value of the optimal control policy converges to the solution of a linear program (LP). We provide necessary and sufficient conditions for a generic control policy to be (i) asymptotically optimal, (ii) asymptotically optimal with square root convergence rate, and (iii) asymptotically optimal with exponential rate. We then construct the LP-index policy that is asymptotically optimal with square root convergence rate on all models and with exponential rate if the model is nondegenerate in finite horizon and satisfies a uniform global attractor property in infinite horizon. We next define the LP-update policy, which is essentially a repeated LP-index policy that solves a new LP at each decision epoch. We conclude by providing numerical experiments to compare the efficiency of different LP-based policies. Funding: This work was supported by Agence Nationale de la Recherche [Grant ANR-19-CE23-0015]. [ABSTRACT FROM AUTHOR]

The exploration of new environments is a crucial challenge for mobile robots. This task becomes even more complex with the added requirement of ensuring safety. Here, safety refers to the robot staying in regions where the values of certain environmental conditions (such as terrain steepness or radiation levels) are within a predefined threshold. We consider two types of safe exploration problems. First, the robot has a map of its workspace, but the values of the environmental features relevant to safety are unknown beforehand and must be explored. Second, both the map and the environmental features are unknown, and the robot must build a map whilst remaining safe. Our proposed framework uses a Gaussian process to predict the value of the environmental features in unvisited regions. We then build a Markov decision process that integrates the Gaussian process predictions with the transition probabilities of the environmental model. The Markov decision process is then incorporated into an exploration algorithm that decides which new region of the environment to explore based on information value, predicted safety, and distance from the current position of the robot. We empirically evaluate the effectiveness of our framework through simulations and its application on a physical robot in an underground environment. [ABSTRACT FROM AUTHOR]

The study addresses the long-term effects of promotions in terms of movement in a value-based segmentation (lead, iron, gold, platinum), instead of simply looking at response rates that occur shortly after the promotion. The study develops a framework for planning an optimal promotion strategy via Markov Decision Processes and Machine Learning methods for an online department store. In the first phase, the states are set as the customer profitability segments in order to conduct the MDPs. Then, MDP model is solved, and the optimal decision for each segment is determined. In the second phase, in order to aid the company for making their plans for the next year, the segment that the customer will belong to next year should be predicted. Prediction of the future customer profitability segment is performed by using several machine learning algorithms, and the best performing model is selected. Using this best performing model, the company can predict the future (potential) profitability segment of the customer and make plans which include the optimal promotions that will be directed to the customers depending on their segments (these optimal promotions are the outcomes of the first phase). The proposed framework can be applied by practitioners in e-commerce companies which keep customer data. [ABSTRACT FROM AUTHOR]

This paper studies constrained Markov decision processes under the total expected discounted cost optimality criterion, with a state-action dependent discount factor that may take any value between zero and one. Both the state and the action space are assumed to be Borel spaces. By using the linear programming approach, consisting in stating the control problem as a linear problem on a set of occupation measures, we show the existence of an optimal stationary Markov policy. Our results are based on the study of both weak-strong topologies in the space of occupation measures and Young measures in the space of Markov policies. [ABSTRACT FROM AUTHOR]

This paper concerns discrete-time infinite-horizon stochastic control systems with Borel state and action spaces and universally measurable policies. We study optimization problems on strategic measures induced by the policies in these systems. The results are then applied to risk-neutral and risk-sensitive Markov decision processes to establish the measurability of the optimal value functions and the existence of universally measurable, randomized or nonrandomized, ϵ-optimal policies, for a variety of average cost criteria and risk criteria. We also extend our analysis to a class of minimax control problems and establish similar optimality results under the axiom of analytic determinacy. Funding: This work was supported by grants from DeepMind, the Alberta Machine Intelligence Institute (AMII), and Alberta Innovates-Technology Futures (AITF). [ABSTRACT FROM AUTHOR]

As an important part of modern transportation, smart rail system need to handle a large number of delay-sensitive and task-intensive tasks in a high-speed mobile state. However, high-speed mobility challenges the traditional information processing modes a lot, such as service interruptions and information congestion. In order to solve these problems, we proposed a service migration strategy based on intelligent agent group cooperative prediction combined with edge computing service migration technology. The aim is to reduce system latency and overhead and ensure service continuity. Firstly, we constructed a cloud-edge-end collaborative scheduling network architecture model for distributed smart rail system is constructed, integrating mobility management and business orchestration to provide effective support for intelligent decision-making. Then we proposed an intelligent grouping collaborative migration strategy by consolidating resources for similar or identical tasks and employing a group negotiation mechanism, where the migration process is divided into four steps: detection, interaction, coordination and execution. Finally, a deep reinforcement learning algorithm is utilized to train multi-agent models for group collaborative prediction in edge migration strategies. The strategy dynamically adjusts task migration between edge nodes and the cloud based on real-time system status and task requirements to optimize its performance. Experimental results show that the proposed architecture and algorithm can effectively reduce total task delay and system overhead. Instead it can guarantee migration rate for task-intensive requirements, and effectively improve the reliability, effectiveness, and safety of smart rail system services. The present study lays a foundation for the future researches on applications of smart rail system. [ABSTRACT FROM AUTHOR]

In recent years, with the accelerated development of technology towards automation and intelligence, autonomous decision-making capabilities in unmanned systems are poised to play a crucial role in contemporary aerial two-player zero-sum games (TZSGs). Deep reinforcement learning (DRL) methods enable agents to make autonomous manoeuvring decisions. This paper focuses on current mainstream DRL algorithms based on fundamental tactical manoeuvres, selecting a typical aerial TZSG scenario—within visual range (WVR) combat. We model the key elements influencing the game using a Markov decision process (MDP) and demonstrate the mathematical foundation for implementing DRL. Leveraging high-fidelity simulation software (Warsim v1.0), we design a prototypical close-range aerial combat scenario. Utilizing this environment, we train mainstream DRL algorithms and analyse the training outcomes. The effectiveness of these algorithms in enabling agents to manoeuvre in aerial TZSG autonomously is summarised, providing a foundational basis for further research. [ABSTRACT FROM AUTHOR]

In this paper we study discrete-time Markov decision processes with Borel state and action spaces under the risk-sensitive average cost criterion. The cost function can be unbounded. We introduce a new kernel and prove the quasi-compactness of the kernel from which the multiplicative Poisson equation is derived. Moreover, we develop a new approach to show the existence of a solution to the risk-sensitive average cost optimality equation and obtain the existence of an optimal deterministic stationary policy. Furthermore, we give two examples to illustrate our results. [ABSTRACT FROM AUTHOR]

We propose a diffusion approximation method to the continuous-state Markov decision processes that can be utilized to address autonomous navigation and control in unstructured off-road environments. In contrast to most decision-theoretic planning frameworks that assume fully known state transition models, we design a method that eliminates such a strong assumption that is often extremely difficult to engineer in reality. We first take the second-order Taylor expansion of the value function. The Bellman optimality equation is then approximated by a partial differential equation, which only relies on the first and second moments of the transition model. By combining the kernel representation of the value function, we design an efficient policy iteration algorithm whose policy evaluation step can be represented as a linear system of equations characterized by a finite set of supporting states. We first validate the proposed method through extensive simulations in 2 D obstacle avoidance and 2.5 D terrain navigation problems. The results show that the proposed approach leads to a much superior performance over several baselines. We then develop a system that integrates our decision-making framework with onboard perception and conduct real-world experiments in both cluttered indoor and unstructured outdoor environments. The results from the physical systems further demonstrate the applicability of our method in challenging real-world environments. [ABSTRACT FROM AUTHOR]

In this paper, we prove the following assertion for an absorbing Markov decision process (MDP) with the given initial distribution, which is also assumed to be semi-continuous: the continuity of the projection mapping from the space of strategic measures to the space of occupation measures, both endowed with their weak topologies, is equivalent to the MDP model being uniformly absorbing. An example demonstrates, among other interesting scenarios, that for an absorbing (but not uniformly absorbing) semi-continuous MDP with the given initial distribution, the space of occupation measures can fail to be compact in the weak topology. [ABSTRACT FROM AUTHOR]

How do you incentivize self-interested agents to explore when they prefer to exploit? We consider complex exploration problems, where each agent faces the same (but unknown) Markov decision process (MDP). In contrast with traditional formulations of reinforcement learning (RL), agents control the choice of policies, whereas an algorithm can only issue recommendations. However, the algorithm controls the flow of information, and can incentivize the agents to explore via information asymmetry. We design an algorithm which explores all reachable states in the MDP. We achieve provable guarantees similar to those for incentivizing exploration in static, stateless exploration problems studied previously. From the RL perspective, we design RL mechanisms, that is, RL algorithms that interact with self-interested agents and are compatible with their incentives. This is the first paper on RL mechanisms, that is, the first paper on any scenario that combines RL and incentives, to the best of our knowledge. How do you incentivize self-interested agents to explore when they prefer to exploit? We consider complex exploration problems, where each agent faces the same (but unknown) Markov decision process (MDP). In contrast with traditional formulations of reinforcement learning, agents control the choice of policies, whereas an algorithm can only issue recommendations. However, the algorithm controls the flow of information, and can incentivize the agents to explore via information asymmetry. We design an algorithm which explores all reachable states in the MDP. We achieve provable guarantees similar to those for incentivizing exploration in static, stateless exploration problems studied previously. To the best of our knowledge, this is the first work to consider mechanism design in a stateful, reinforcement learning setting. Supplemental Material: The online appendix is available at https://doi.org/10.1287/opre.2022.0495. [ABSTRACT FROM AUTHOR]

This paper addresses a class of discrete-time Markov decision processes in Borel spaces with a finite number of cost constraints. The constrained control model considers costs of discounted type with state-dependent discount factors which are subject to external disturbances. Our objective is to prove the existence of optimal control policies and characterize them according to certain optimality criteria. Specifically, by rewriting appropriately our original constrained problem as a new one on a space of occupation measures, we apply the direct method to show solvability. Next, the problem is defined as a convex program, and we prove that the existence of a saddle point of the associated Lagrangian operator is equivalent to the existence of an optimal control policy for the constrained problem. Finally, we turn our attention to multi-objective optimization problems, where the existence of Pareto optimal policies can be obtained from the existence of saddle-points of the aforementioned Lagrangian or equivalently from the existence of optimal control policies of constrained problems. [ABSTRACT FROM AUTHOR]

Controlled discrete time Markov processes are studied first with long run general discounting functional. It is shown that optimal strategies for average reward per unit time problem are also optimal for average generally discounting functional. Then long run risk sensitive reward functional with general discounting is considered. When risk factor is positive then optimal value of such reward functional is dominated by the reward functional corresponding to the long run risk sensitive control. In the case of negative risk factor we get an asymptotical result, which says that optimal average reward per unit time control is nearly optimal for long run risk sensitive reward functional with general discounting, assuming that risk factor is close to 0. For this purpose we show in Appendix upper estimates for large deviations of weighted empirical measures, which are of independent interest. [ABSTRACT FROM AUTHOR]

Black-box systems are inherently hard to verify. Many verification techniques, like model checking, require formal models as a basis. However, such models often do not exist, or they might be outdated. Active automata learning helps to address this issue by offering to automatically infer formal models from system interactions. Hence, automata learning has been receiving much attention in the verification community in recent years. This led to various efficiency improvements, paving the way toward industrial applications. Most research, however, has been focusing on deterministic systems. In this article, we present an approach to efficiently learn models of stochastic reactive systems. Our approach adapts L ∗ -based learning for Markov decision processes, which we improve and extend to stochastic Mealy machines. When compared with previous work, our evaluation demonstrates that the proposed optimizations and adaptations to stochastic Mealy machines can reduce learning costs by an order of magnitude while improving the accuracy of learned models. [ABSTRACT FROM AUTHOR]

The Stochastic Shortest Path (SSP) is a formalism widely used for modeling goal-oriented probabilistic planning problems. When dead ends, which are states from which goal states cannot be reached, are present in the problem and cannot be avoided, the standard criterion for solving SSPs is not well defined in these scenarios. Because of that, several alternate criteria for solving SSPs with unavoidable dead ends have been proposed in the literature. One of these criteria is GUBS (Goals with Utility-Based Semantics), a criterion that makes trade-offs between probability-to-goal and cost by combining goal prioritization with Expected Utility Theory. GUBS is a good choice for these problems because it is one of the only criteria that are known to maintain the ?-strong probability-to-goal priority property, a property that provides guarantees on how a decision criterion can choose policies without having to preprocess any specific SSP problem. Although there already exist two exact algorithms for solving GUBS, eGUBS-VI and eGUBS-AO*, both are offline and there is no algorithm for solving GUBS in an online manner. In this paper we propose UCT-GUBS, an online approximate algorithm based on UCT (a Monte Carlo tree search algorithm) that solves SSPs under the GUBS criterion. We provide an analysis of an empirical evaluation performed on two probabilistic planning domains (Triangle Tireworld and Navigation) to observe how the probability-to-goal and utility values of the resulting policies compare to the optimal values, and also how the time performance of UCT-GUBS compares to the ones of eGUBS-VI and eGUBS-AO*. Our conclusion is that, like other algorithms, the usage of UCT-GUBS has to be evaluated considering the application requirements and of the problem being solved. Depending on these factors, it can be a good alternative for obtaining policies in an online fashion while, for some problems, also being able to have better time performance than other algorithms

This paper develops a sequential decision-making framework called constrained restless multi-armed bandits (CRMABs) to model problems of resource allocation under uncertainty and dynamic availability constraints. The decision-maker’s objective is to maximize the long-term cumulative reward. This can only be achieved by considering the impact of current actions on the future evolution of states. The uncertainty about the future availability of arms and partial state-information makes this objective challenging. CRMABs can be applied to resource allocation problems in cyber-physical systems, including sensor/relay scheduling. Whittle’s index policy, online rollout, and myopic policies are studied as solutions for CRMABs. First, the conditions for the applicability of Whittle’s index policy are studied, and the indexability result is claimed under some structural assumptions. An algorithm for index computation is presented. The online rollout policy for partially observable CRMABs is proposed as a low-complexity alternative to the index policy, and the complexity of these schemes is analyzed. An upper bound on the optimal value function is derived, which helps assess the sub-optimality of various solutions. The simulation study compares the performance of these policies and shows that the rollout policy is the better performing solution. In some settings it shows about 14% gain relative to Whittle’s index and myopic policies. Finally, an application to the problem of wildfire management is presented. Decision-making using CRMABs is analyzed from the perspective of a central agency tasked with fighting wildfires in multiple regions under logistic constraints.

To enhance the operational safety and efficiency of aircraft under uncertain or unknown flight intentions, a decision-making framework based on Markov Decision Processes with incomplete information (IIG-MDP) is proposed in this paper. Firstly, the paper incorporates the potential impact of the target aircraft and its surrounding traffic flow into the decision-making assessment, calculating the components of the decision model’s state space, among other elements. Secondly, the continuous episode is discretized into a multi- episode space, transforming the interval maintenance problem into a discrete multi-episode decision-making problem, and at the beginning of each episode, the action estimate for the target is corrected based on the observed state of the target. Thirdly, an episode is further discretized into a series of decision moments, and an n-Step Approximate Dynamic Programming (ADP) algorithm is proposed to calculate the payoff value of action strategies at each decision moment, obtaining the optimal decision sequence within an episode, and then updating the initial state of the next episode for iterative calculation until the end of the flight. Through simulation experiments, the IIG-MDP model and algorithm are verified, and the results show that the 3-Step ADP algorithm used in this paper can significantly reduce the computational dimension of the dynamic programming method, improving computational efficiency. Compared with the Monte Carlo and MPC decision-making models, it offers better decision choices while ensuring safety. Due to the prediction and updating of the target’s state, this method also has better real-time performance and practicality.

Multivariate time series (MTS) forecasting is a rapidly expanding field with diverse and futuristic applications. However, traditional statistical learning models need more prediction accuracy when faced with dynamic variability, non-linearity, and non-stationarity, as well as the challenge of selecting MTS data for classification. Moreover, the existing methods for early classification of multivariate time series data suffer from numerous severe challenges, including evaluating the length of testing the MTS data component, which must be equal to the training MTS data component, and the availability of faulty data components in MTS. To address this issue, we propose a novel framework for early MTS forecasting using double Q-learning-based ensemble techniques to improve prediction accuracy. Our framework uses Q-learning agents to select optimal actions, which results in maximum rewards and accurate prediction. We investigate the ensemble behavior of learned agents using double Q-learning and Gaussian Process Classifiers (GPC) for early forecasting of MTS data. We also determine the minimum required time-series length for classifying faulty data components using the probabilistic Auto-Regressive Integrated Moving Average (ARIMA) model, enhancing framework robustness and mitigating miss-classification accuracy. Our proposed framework achieves 99.89% accuracy for early forecasting, surpassing existing methods based on different benchmark settings and publicly available multivariate time-series datasets. The framework provides a promising solution to the challenges of accurate MTS forecasting and offers insights into the early prediction of recent stock market trading data.

Background: Current obstructive sleep apnea treatment relies on manual PAP titration, but it has limitations. Complex interactions during titration and variations in SpO2 data accuracy pose challenges. Patients with co-occurring chronic hypercapnia may require precise oxygen titration. To address these issues, we propose a Clinical Decision Support System using Markov decision processes. Methods: This study, compliant with data protection laws, focused on adults with OSA-induced hypoxemia utilizing supplemental oxygen and CPAP/BiPAP therapy. PAP titration, conducted over one night, involved vigilant monitoring of vital signs and physiological parameters. Adjustments to CPAP pressure, potential BiLevel transitions, and supplemental oxygen were precisely guided by patient metrics. Markov decision processes outlined three treatment actions for disorder management, incorporating expert medical insights. Results: In our study involving 14 OSA patients (average age: 63 years, 27% females, BMI 41 kg m−2), significant improvements were observed in key health parameters after manual titration. The initial AHI of 61.8 events per hour significantly decreased to an average of 18.0 events per hour after PAP and oxygen titration (p < 0.0001), indicating a substantial reduction in sleep-disordered breathing severity. Concurrently, SpO2 levels increased significantly from an average of 79.7% before titration to 89.1% after titration (p < 0.0003). Pearson correlation coefficients demonstrated aggravation of hypercapnia in 50% of patients (N = 5) with initial pCO2 < 55 mmHg during the increase in CPAP pressure. However, transitioning to BiPAP exhibited a reduction in pCO2 levels, showcasing its efficacy in addressing hypercapnia. Simultaneously, BiPAP therapy correlated with a substantial increase in SpO2, underscoring its positive impact on oxygenation in OSA patients. Markov Decision Process analysis demonstrated realistic patient behavior during stable night conditions, emphasizing minimal apnea and good toleration to high CPAP pressure. Conclusions: The development of a framework for Markov decision processes of PAP and oxygen titration algorithms holds promise for providing algorithms for improving pCO2 and SpO2 values. While challenges remain, including the need for high-quality data, the potential benefits in terms of patient management and care optimization are substantial, and this approach represents an exciting frontier in the realm of telemedicine and respiratory healthcare. [ABSTRACT FROM AUTHOR]

We consider a two-stage service system with two types of servers, namely subordinates who perform the first-stage service and supervisors who have their own responsibilities in addition to collaborating with the subordinates on the second-stage service. Rewards are earned when first- or second-stage service is completed and when supervisors finish one of their own responsibilities. Costs are incurred when impatient customers abandon without completing the second-stage service. Our problem is to determine how the supervisors should distribute their time between their joint work with the subordinates and their own responsibilities. Under the assumptions that service times at both stages are exponentially distributed and that the customers waiting for second-stage service abandon after an exponential amount of time, we prove that one of two policies will maximize the long-run average profit. Namely, it is optimal for supervisors to start collaborating with subordinates either when subordinates can no longer serve new customers or as soon as there is a customer ready for second-stage service. Furthermore, we show that the optimality condition is a simple threshold on the system parameters. We conclude by proving that pooling supervisors (and their associated subordinates) improves system performance, but with limited returns as more supervisors are pooled. [ABSTRACT FROM AUTHOR]

We develop a regression-based primal-dual martingale approach for solving discrete time, finite-horizon MDPs. The state and action spaces may be finite or infinite (but regular enough) subsets of Euclidean space. Consequently, our method allows for the construction of tight upper and lower-biased approximations of the value functions, providing precise estimates of the optimal policy. Importantly, we prove error bounds for the estimated duality gap featuring polynomial dependence on the time horizon. Additionally, we observe sublinear dependence of the stochastic part of the error on the cardinality/dimension of the state and action spaces. From a computational perspective, our proposed method is efficient. Unlike typical duality-based methods for optimal control problems in the literature, the Monte Carlo procedures involved here do not require nested simulations. [ABSTRACT FROM AUTHOR]

Problem definition: Effective hypertension management is critical to reducing the consequences of atherosclerotic cardiovascular disease, a leading cause of death in the United States. Clinical guidelines for hypertension can be enhanced using decision-analytic approaches capable of capturing complexities in treatment planning. However, model-generated recommendations may be uninterpretable/unintuitive, limiting their clinical acceptability. We address this challenge by investigating interpretable treatment plans. Methodology/results: We formulate interpretable treatment plans as Markov decision processes (MDPs) and analyze the problems of optimizing monotone policies, which prohibit decreasing treatment intensity for sicker patients, and class-ordered monotone policies, which generalize monotone policies. We establish that both policies depend on initial state distributions and that optimal monotone policies can be generated tractably for many treatment planning problems. Next, we propose exact formulations for optimizing interpretable policies broadly. Then, we analyze the price of interpretability, proving that the class-ordered monotone policy's price of interpretability does not exceed the monotone policy's price of interpretability. Finally, we formulate and evaluate MDPs for hypertension treatment planning using a large nationally representative data set of the U.S. population. We compare the structure and performance of optimal monotone policies and class-ordered monotone policies with optimal MDP-based policies and current clinical guidelines. At the patient level, optimal MDP-based policies may be unintuitive, recommending more aggressive treatment for healthier patients than sicker patients. Conversely, monotone policies and class-ordered monotone policies never deescalate treatment, reflecting clinical intuition. Across 66.5 million patients, optimized monotone policies and class-ordered monotone policies outperform clinical guidelines, saving over 3,246 quality-adjusted life years per 100,000 patients, with both policies paying a low price of interpretability. Sensitivity analysis illustrates that monotone policies and class-ordered monotone policies are robust to various definitions of "interpretability." Managerial implications: Interpretable policies can be tractably optimized, drastically outperform existing guidelines, and perform near optimally—potentially increasing the acceptability of decision-analytic approaches in practice. Funding: L. N. Steimle and W. J. Marrero received support from the National Science Foundation Graduate Research Fellowship [Grant DGE 1256260]. J. B. Sussman received support from the National Institutes of Health [Grants R01NS102715 and RF1AG068410], the U.S. Department of Veterans Affairs [Grants 1I01-HX003304 and 1I50-HX003251], and the Michigan Department of Health and Human Services. Supplemental Material: The e-companion is available at https://doi.org/10.1287/msom.2021.0373. [ABSTRACT FROM AUTHOR]

In this paper, we present a brief survey of reinforcement learning, with particular emphasis on stochastic approximation (SA) as a unifying theme. The scope of the paper includes Markov reward processes, Markov decision processes, SA algorithms, and widely used algorithms such as temporal difference learning and Q-learning.