Search

As a primary contribution, we present a convergence theorem for stochastic iterations, and in particular, Q-learning iterates, under a general, possibly non-Markovian, stochastic environment. Our conditions for convergence involve an ergodicity and a positivity criterion. We provide a precise characterization on the limit of the iterates and conditions on the environment and initializations for convergence. As our second contribution, we discuss the implications and applications of this theorem to a variety of stochastic control problems with non-Markovian environments involving (i) quantized approximations of fully observed Markov Decision Processes (MDPs) with continuous spaces (where quantization break down the Markovian structure), (ii) quantized approximations of belief-MDP reduced partially observable MDPS (POMDPs) with weak Feller continuity and a mild version of filter stability (which requires the knowledge of the model by the controller), (iii) finite window approximations of POMDPs under a uniform controlled filter stability (which does not require the knowledge of the model), and (iv) for multi-agent models where convergence of learning dynamics to a new class of equilibria, subjective Q-learning equilibria, will be studied. In addition to the convergence theorem, some implications of the theorem above are new to the literature and others are interpreted as applications of the convergence theorem. Some open problems are noted., Comment: 2 figures

We study mean-field control problems in discrete-time under the infinite horizon average cost optimality criteria. We focus on both the finite population and the infinite population setups. We show the existence of a solution to the average cost optimality equation (ACOE) and the existence of optimal stationary Markov policies for finite population problems under (i) a minorization condition that provides geometric ergodicity on the collective state process of the agents, and (ii) under standard Lipschitz continuity assumptions on the stage-wise cost and transition function of the agents when the Lipschitz constant of the transition function satisfies a certain bound. For the infinite population problem, we establish the existence of a solution to the ACOE, and the existence of optimal policies under the continuity assumptions on the cost and the transition functions. Finally, we relate the finite population and infinite population control problems: (i) we prove that the optimal value of the finite population problem converges to the optimal value of the infinite population problem as the number of agents grows to infinity; (ii) we show that the accumulation points of the finite population optimal solution corresponds to an optimal solution for the infinite population problem, and finally (iii), we show that one can use the solution of the infinite population problem for the finite population problem symmetrically across the agents to achieve near optimal performance when the population is sufficiently large.

For infinite-horizon average-cost criterion problems, there exist relatively few rigorous approximation and reinforcement learning results. In this paper, for such problems, we present several approximation and reinforcement learning results for Markov Decision Processes with standard Borel spaces. Toward this end, (i) we first provide a discretization based approximation method for fully observed Markov Decision Processes (MDPs) with continuous spaces under average cost criteria, and we provide error bounds for the approximations when the dynamics are only weakly continuous (for asymptotic convergence of errors as the grid sizes vanish) or Wasserstein continuous (with a rate in approximation as the grid sizes vanish) under certain ergodicity assumptions. In particular, we relax the total variation condition given in prior work to weak continuity as well as Wasserstein continuity conditions. (ii) We provide synchronous and asynchronous Q-learning algorithms for continuous spaces via quantization (where the quantized state is taken to be the actual state in corresponding Q-learning algorithms presented in the paper), and establish their convergence; for the former we utilize a span semi-norm approach and for the latter we use a direct contraction approach. (iii) We finally show that the convergence is to the optimal Q values of the finite approximate models constructed via quantization, which implies near optimality of the arrived solution. Our Q-learning convergence results and their convergence to near optimality are new for continuous spaces, and the proof method is new even for finite spaces, to our knowledge., Comment: 3 figures

Linear Wireless Sensor Networks (LWSNs) are used in applications where deployment scenarios necessitate sensor nodes to be placed over a line topology. However, such a deployment raises reliability concerns because almost all the nodes in the network are critical with respect to the survivability of the LWSN. It is possible that an LWSN can stay connected even if a subset of the nodes are eliminated, yet, the potential reduction in Network Lifetime (NL) due to such an occurrence can be significant. In this study, after presenting a concise survey of the literature on LWSN reliability, we present an elaborate optimization framework to model the operation of an LWSN, which is built upon a comprehensive system model. Our framework encompasses three transmission power and packet size assignment strategies, which are instrumental in characterizing LWSN behavior. Furthermore, we utilized two-node failure models (i.e., random and coordinated) to assess the vulnerability of LWSNs from multiple perspectives. The results of this study reveal that the impact of coordinated node failures on NL is more severe than the impact of random node failures to such extent that in strongly connected LWSNs, the percentage decrease in NL due to coordinated node failures can be more than a magnitude higher than the NL decrease due to random node failures.

Linear Wireless Sensor Networks (LWSNs) are used in applications where deployment scenarios necessitate sensor nodes to be placed over a line topology. However, such a deployment raises reliability concerns because almost all the nodes in the network are critical with respect to the survivability of the LWSN. It is possible that an LWSN can stay connected even if a subset of the nodes are eliminated, yet, the potential reduction in Network Lifetime (NL) due to such an occurrence can be significant. In this study, after presenting a concise survey of the literature on LWSN reliability, we present an elaborate optimization framework to model the operation of an LWSN, which is built upon a comprehensive system model. Our framework encompasses three transmission power and packet size assignment strategies, which are instrumental in characterizing LWSN behavior. Furthermore, we utilized two-node failure models (i.e., random and coordinated) to assess the vulnerability of LWSNs from multiple perspectives. The results of this study reveal that the impact of coordinated node failures on NL is more severe than the impact of random node failures to such extent that in strongly connected LWSNs, the percentage decrease in NL due to coordinated node failures can be more than a magnitude higher than the NL decrease due to random node failures.

Reinforcement learning algorithms often require finiteness of state and action spaces in Markov decision processes (MDPs) (also called controlled Markov chains) and various efforts have been made in the literature towards the applicability of such algorithms for continuous state and action spaces. In this paper, we show that under very mild regularity conditions (in particular, involving only weak continuity of the transition kernel of an MDP), Q-learning for standard Borel MDPs via quantization of states and actions (called Quantized Q-Learning) converges to a limit, and furthermore this limit satisfies an optimality equation which leads to near optimality with either explicit performance bounds or which are guaranteed to be asymptotically optimal. Our approach builds on (i) viewing quantization as a measurement kernel and thus a quantized MDP as a partially observed Markov decision process (POMDP), (ii) utilizing near optimality and convergence results of Q-learning for POMDPs, and (iii) finally, near-optimality of finite state model approximations for MDPs with weakly continuous kernels which we show to correspond to the fixed point of the constructed POMDP. Thus, our paper presents a very general convergence and approximation result for the applicability of Q-learning for continuous MDPs.

In this paper, for POMDPs, we provide the convergence of a Q learning algorithm for control policies using a finite history of past observations and control actions, and, consequentially, we establish near optimality of such limit Q functions under explicit filter stability conditions. We present explicit error bounds relating the approximation error to the length of the finite history window. We establish the convergence of such Q-learning iterations under mild ergodicity assumptions on the state process during the exploration phase. We further show that the limit fixed point equation gives an optimal solution for an approximate belief-MDP. We then provide bounds on the performance of the policy obtained using the limit Q values compared to the performance of the optimal policy for the POMDP, where we also present explicit conditions using recent results on filter stability in controlled POMDPs. While there exist many experimental results, (i) the rigorous asymptotic convergence (to an approximate MDP value function) for such finite-memory Q-learning algorithms, and (ii) the near optimality with an explicit rate of convergence (in the memory size) are results that are new to the literature, to our knowledge.

We explored the joint effect of synchronization window and offset/drift mode selection on the time synchronization of linear wireless sensor networks (LWSNs). Recent advances in the field along with the availability of capable hardware led to adoption of LWSNs in diverse areas like monitoring of roads, pipelines, and tunnels. The linear topology applications are susceptible to single point of failure; therefore, energy efficient operation of LWSNs is even more important than the traditional WSNs. To address the challenge, we investigate the time synchronization mode selection for the optimum operation of a multi-hop and low-overhead LWSN. We investigate two modes of synchronization: synchronization by using only offset and synchronization by using offset in addition to the clock drift. Furthermore, we investigate the effects of synchronization window size. Our experimental results reveal that computation of offset alone for smaller window sizes and resynchronization periods is sufficient in achieving acceptable degree of synchronization.

We explored the joint effect of synchronization window and offset/drift mode selection on the time synchronization of linear wireless sensor networks (LWSNs). Recent advances in the field along with the availability of capable hardware led to adoption of LWSNs in diverse areas like monitoring of roads, pipelines, and tunnels. The linear topology applications are susceptible to single point of failure; therefore, energy efficient operation of LWSNs is even more important than the traditional WSNs. To address the challenge, we investigate the time synchronization mode selection for the optimum operation of a multi-hop and low-overhead LWSN. We investigate two modes of synchronization: synchronization by using only offset and synchronization by using offset in addition to the clock drift. Furthermore, we investigate the effects of synchronization window size. Our experimental results reveal that computation of offset alone for smaller window sizes and resynchronization periods is sufficient in achieving acceptable degree of synchronization.

We study continuity and robustness properties of infinite-horizon average expected cost problems with respect to (controlled) transition kernels, and applications of these results to the problem of robustness of control policies designed for approximate models applied to actual systems. We show that sufficient conditions presented in the literature for discounted-cost problems are in general not sufficient to ensure robustness for average-cost problems. However, we show that the average optimal cost is continuous in the convergences of controlled transition kernel models where convergence of models entails (i) continuous weak convergence in state and actions, and (ii) continuous setwise convergence in the actions for every fixed state variable, in addition to either uniform ergodicity or some regularity conditions. We establish that the mismatch error due to the application of a control policy designed for an incorrectly estimated model to the true model decreases to zero as the incorrect model approaches the true model under the stated convergence criteria. Our findings significantly relax related studies in the literature which have primarily considered the more restrictive total variation convergence criteria. Applications to robustness to models estimated through empirical data (where almost sure weak convergence criterion typically holds, but stronger criteria do not) are studied and conditions for asymptotic robustness to data-driven learning are established., Comment: Presented at Conference on Decision and Control 2019. arXiv admin note: text overlap with arXiv:1803.06046

This study exhibits relationships between length-weight and length-length for 21 fish species caught in Izmir Bay. Coefficients a and b for the length-weight relationships (LWRs) and length-length relationships (LLRs) were calculated with W=aLb formulae and as equations of TL=a+bFL and TL=a+bSL respectively. Equations of length-length for converting standard length and fork length into total length and vice versa were proven linear. The involved relationships were significantly interrelated (R2>0.942). Values b in the LWRs varied from 2.21 to 3.96 (mean ± SE: 3.15 ± 0.08). The student’s t-test showed that 87.9% of values b were significantly different from 3., U ovom radu se prikazuje povezanost dužinsko-masenog odnosa i dužinsko-dužinskog odnosa za 21 vrstu riba ulovljenih u Izmirskom zaljevu. Koeficijenti a i b za odnos dužinsko-masenog (LWR) i odnos dužinsko-dužinskog (LLR) izračunati su s formulama W = a/b i kao jednadžbom TL = a + bFL i TL = a + bSL. Jednadžbe za pretvaranja standardne duljine i vilične duljine u ukupnu duljinu i obrnuto, pokazale su se linearnima. Uključeni odnosi bili su značajno međusobno povezani (R2> 0,942). Vrijednosti b u LWR varirale su od 2,21 do 3,96 (srednja vrijednost ± SE: 3,15 ± 0,08). Stu-dentov t-test pokazao je da se 87,9% vrijednosti b značajno razlikuje od 3.

In the theory of Partially Observed Markov Decision Processes (POMDPs), existence of optimal policies have in general been established via converting the original partially observed stochastic control problem to a fully observed one on the belief space, leading to a belief-MDP. However, computing an optimal policy for this fully observed model, and so for the original POMDP, using classical dynamic or linear programming methods is challenging even if the original system has finite state and action spaces, since the state space of the fully observed belief-MDP model is always uncountable. Furthermore, there exist very few rigorous value function approximation and optimal policy approximation results, as regularity conditions needed often require a tedious study involving the spaces of probability measures leading to properties such as Feller continuity. In this paper, we study a planning problem for POMDPs where the system dynamics and measurement channel model are assumed to be known. We construct an approximate belief model by discretizing the belief space using only finite window information variables. We then find optimal policies for the approximate model and we rigorously establish near optimality of the constructed finite window control policies in POMDPs under mild non-linear filter stability conditions and the assumption that the measurement and action sets are finite (and the state space is real vector valued). We also establish a rate of convergence result which relates the finite window memory size and the approximation error bound, where the rate of convergence is exponential under explicit and testable exponential filter stability conditions. While there exist many experimental results and few rigorous asymptotic convergence results, an explicit rate of convergence result is new in the literature, to our knowledge.

Electronic Support Measures (ESM) systems have many operational challenges while locating radar emitter’s position around irregular terrains such as islands due to multipath scattering. To overcome these challenges, this paper addresses exploiting multipath scattering in passive localization of radar emitters around irregular terrains. The idea is based on the use of multipath scattered signals as virtual sensor through Geographical Information System (GIS). In this way, it is presented that single receiver (ESM receiver) passive localization can be achieved for radar emitters. The study is initiated with estimating candidate multipath scattering centers over irregular terrain. To do this, ESM receivers’ Angle of Arrival (AOA) and Time of Arrival (TOA) information are required for directly received radar pulses along with multipath scattered pulses. The problem then turns out to be multiple-sensor localization problem for which Time Difference of Arrival (TDOA)-based techniques can easily be applied. However, there is high degree of uncertainty in location of candidate multipath scattering centers as the multipath scattering involves diffuse components over irregular terrain. Apparently, this causes large localization errors in TDOA. To reduce this error, a reliability based weighting method is proposed. Simulation results regarding with a simplified 3D model is also presented.

We present a new point-based approach for modeling the appearance of real scenes. The approach uses a raw point cloud as the geometric representation of a scene, and augments each point with a learnable neural descriptor that encodes local geometry and appearance. A deep rendering network is learned in parallel with the descriptors, so that new views of the scene can be obtained by passing the rasterizations of a point cloud from new viewpoints through this network. The input rasterizations use the learned descriptors as point pseudo-colors. We show that the proposed approach can be used for modeling complex scenes and obtaining their photorealistic views, while avoiding explicit surface estimation and meshing. In particular, compelling results are obtained for scene scanned using hand-held commodity RGB-D sensors as well as standard RGB cameras even in the presence of objects that are challenging for standard mesh-based modeling.

We present a system for learning full-body neural avatars, i.e. deep networks that produce full-body renderings of a person for varying body pose and camera position. Our system takes the middle path between the classical graphics pipeline and the recent deep learning approaches that generate images of humans using image-to-image translation. In particular, our system estimates an explicit two-dimensional texture map of the model surface. At the same time, it abstains from explicit shape modeling in 3D. Instead, at test time, the system uses a fully-convolutional network to directly map the configuration of body feature points w.r.t. the camera to the 2D texture coordinates of individual pixels in the image frame. We show that such a system is capable of learning to generate realistic renderings while being trained on videos annotated with 3D poses and foreground masks. We also demonstrate that maintaining an explicit texture representation helps our system to achieve better generalization compared to systems that use direct image-to-image translation.

In stochastic control applications, typically only an ideal model (controlled transition kernel) is assumed and the control design is based on the given model, raising the problem of performance loss due to the mismatch between the assumed model and the actual model. Toward this end, we study continuity properties of discrete-time stochastic control problems with respect to system models (i.e., controlled transition kernels) and robustness of optimal control policies designed for incorrect models applied to the true system. We study both fully observed and partially observed setups under an infinite horizon discounted expected cost criterion. We show that continuity and robustness cannot be established under weak and setwise convergences of transition kernels in general, but that the expected induced cost is robust under total variation. By imposing further assumptions on the measurement models and on the kernel itself (such as continuous convergence), we show that the optimal cost can be made continuous under weak convergence of transition kernels as well. Using these continuity properties, we establish convergence results and error bounds due to mismatch that occurs by the application of a control policy which is designed for an incorrectly estimated system model to a true model, thus establishing positive and negative results on robustness.Compared to the existing literature, we obtain strictly refined robustness results that are applicable even when the incorrect models can be investigated under weak convergence and setwise convergence criteria (with respect to a true model), in addition to the total variation criteria. These entail positive implications on empirical learning in (data-driven) stochastic control since often system models are learned through empirical training data where typically weak convergence criterion applies but stronger convergence criteria do not., Comment: Conference version to appear at the 2018 IEEE CDC with title "Robustness to Incorrect System Models in Stochastic Control and Application to Data-Driven Learning". The paper is to appear in SIAM J. on Control and Optimization

We study the continuity properties of optimal solutions to stochastic control problems with respect to initial probability measures and applications of these to the robustness of optimal control policies applied to systems with incomplete or incorrect priors. It is shown that for single and multi-stage optimal cost problems, continuity and robustness cannot be established under weak convergence or Wasserstein convergence in general, but that the optimal cost is continuous in the priors under the convergence in total variation under mild conditions. By imposing further assumptions on the measurement models, robustness and continuity also hold under weak convergence of priors. We thus obtain robustness results and bounds on the mismatch error that occurs due to the application of a control policy which is designed for an incorrectly estimated prior in terms of a distance measure between the true prior and the incorrect one. Positive and negative practical implications of these results in empirical learning for stochastic control will be presented, where almost surely weak convergence of i.i.d. empirical measures occurs but stronger notions of convergence, such as total variation convergence, in general, do not.

Ilknur Erdem,1 Senay Elbasan Omar,1 Ridvan Kara Ali,1 Hayati Gunes,2 Aynur Eren Topkaya2 1Department of Infectious Diseases, 2Department of Medical Microbiology, Faculty of Medicine, Namik Kemal University, Tekirdag, Turkey Objective: Infections are among the most important causes of morbidity and mortality in patients with systemic lupus erythematosus (SLE) but are rare initial presentation of the disease. Therefore, in this study, we describe a case of Streptococcus pneumoniae sepsis in a young woman with previously undiagnosed SLE. Case report: A 23-year-old female patient was admitted to our outpatient clinic complaining of high fever (40°C), chills, fatigue, generalized myalgia, and cough with brown sputum for 5 days. Blood cultures grew gram-positive coccus defined as S. pneumoniae using standard procedures. Antinuclear antibody was positive at a titer of 1/1,000, and anti-double-stranded DNA was positive at 984 IU/mL. She was diagnosed with SLE. Her respiratory symptoms and pleural effusion were considered to be due to pulmonary manifestation of SLE. Conclusion: The underlying immunosuppression caused by SLE could have predisposed the patient to invasive pneumococcal disease. It may also occur as a primary presenting feature, although a rare condition. Keywords: Streptococcus pneumoniae, sepsis, systemic lupus erythematosus

Aims: Our aim was to report on a survey initiated by the European Association of Percutaneous Cardiovascular Interventions (EAPCI) concerning opinion on the evidence relating to dual antiplatelet therapy (DAPT) duration after coronary stenting. Methods and results: Results from three randomised clinical trials were scheduled to be presented at the American Heart Association Scientific Sessions 2014 (AHA 2014). A web-based survey was distributed to all individuals registered in the EuroIntervention mailing list (n=15,200) both before and after AHA 2014. A total of 1,134 physicians responded to the first (i.e., before AHA 2014) and 542 to the second (i.e., after AHA 2014) survey. The majority of respondents interpreted trial results consistent with a substantial equipoise regarding the benefits and risks of an extended versus a standard DAPT strategy. Two respondents out of ten believed extended DAPT should be implemented in selected patients. After AHA 2014, 46.1% of participants expressed uncertainty about the available evidence on DAPT duration, and 40.0% the need for clinical guidance. Conclusions: This EAPCI survey highlights considerable uncertainty within the medical community with regard to the optimal duration of DAPT after coronary stenting in the light of recent reported trial results. Updated recommendations for practising physicians to guide treatment decisions in routine clinical practice should be provided by international societies.

The objective of this study is to investigate the characteristics and behavioral tendencies of Swedish consumers in relation to their involvement with various types of retail stores as well as their shopping and patronizing behaviors under varying circumstances. In particular, the study focuses on examining the underlying dimensions of Swedish consumers' retail shopping behavior incorporating their lifestyles and attitudinal orientations as well as demographic and socio-economic characteristics. Data for the study were collected through personal interviews in Sweden. The study findings indicate that the most dominant lifestyle dimensions include fashion orientation, adventure orientation and aesthetic orientation. Several of these lifestyle dimensions apparent among the Swedish retail shoppers were found to be related to the number of stores shopped in a given period of time. Also, the most dominant factors were one-stop shopping, merchandise quality and assortment, and store communication factors. When the dimensions extracted based on the grocery store choice are used in explaining the number of stores visited, then we see an inverse relationship between the number of stores visited and the factors of one-stop shopping, merchandise quality, and poverty of time. On the other hand, there is a direct relationship between the number of stores visited and the factor of store communication and convenience. The findings of the study offer important research, public policy and managerial implications for retailer distributors and consumer goods manufacturers.