Search

Efficient utilization of satellite resources in dynamic environments remains a challenging problem in satellite scheduling. This paper addresses the multi-satellite collection scheduling problem (m-SatCSP), aiming to optimize task scheduling over a constellation of satellites under uncertain conditions such as cloud cover. Leveraging Monte Carlo Tree Search (MCTS), a stochastic search algorithm, two versions of MCTS are explored to schedule satellites effectively. Hyperparameter tuning is conducted to optimize the algorithm's performance. Experimental results demonstrate the effectiveness of the MCTS approach, outperforming existing methods in both solution quality and efficiency. Comparative analysis against other scheduling algorithms showcases competitive performance, positioning MCTS as a promising solution for satellite task scheduling in dynamic environments., Comment: 11 pages, 4 figures

In reinforcement learning, agents have successfully used environments modeled with Markov decision processes (MDPs). However, in many problem domains, an agent may suffer from noisy observations or random times until its subsequent decision. While partially observable Markov decision processes (POMDPs) have dealt with noisy observations, they have yet to deal with the unknown time aspect. Of course, one could discretize the time, but this leads to Bellman's Curse of Dimensionality. To incorporate continuous sojourn-time distributions in the agent's decision making, we propose that partially observable semi-Markov decision processes (POSMDPs) can be helpful in this regard. We extend \citet{Spaan2005a} randomized point-based value iteration (PBVI) \textsc{Perseus} algorithm used for POMDP to POSMDP by incorporating continuous sojourn time distributions and using importance sampling to reduce the solver complexity. We call this new PBVI algorithm with importance sampling for POSMDPs -- \textsc{ChronosPerseus}. This further allows for compressed complex POMDPs requiring temporal state information by moving this information into state sojourn time of a POMSDP. The second insight is that keeping a set of sampled times and weighting it by its likelihood can be used in a single backup; this helps further reduce the algorithm complexity. The solver also works on episodic and non-episodic problems. We conclude our paper with two examples, an episodic bus problem and a non-episodic maintenance problem., Comment: 33 pages, 9 figures

The technologies used in smart homes have recently improved to learn the user preferences from feedback in order to enhance the user convenience and quality of experience. Most smart homes learn a uniform model to represent the thermal preferences of users, which generally fails when the pool of occupants includes people with different sensitivities to temperature, for instance due to age and physiological factors. Thus, a smart home with a single optimal policy may fail to provide comfort when a new user with a different preference is integrated into the home. In this paper, we propose a Bayesian Reinforcement learning framework that can approximate the current occupant state in a partially observable smart home environment using its thermal preference, and then identify the occupant as a new user or someone is already known to the system. Our proposed framework can be used to identify users based on the temperature and humidity preferences of the occupant when performing different activities to enable personalization and improve comfort. We then compare the proposed framework with a baseline long short-term memory learner that learns the thermal preference of the user from the sequence of actions which it takes. We perform these experiments with up to 5 simulated human models each based on hierarchical reinforcement learning. The results show that our framework can approximate the belief state of the current user just by its temperature and humidity preferences across different activities with a high degree of accuracy.

Animal interval timing is often studied through the peak interval (PI) procedure. In this procedure, the animal is rewarded for the first response after a fixed delay from the stimulus onset, but on some trials, the stimulus remains and no reward is given. The common methods and models to analyse the response pattern describe it as break-run-break, a period of low rate response followed by rapid responding, followed by a low rate of response. The study of the pattern has found correlations between start, stop, and duration of the run period that hold across species and experiment. It is commonly assumed that in order to achieve the statistics with a pacemaker accumulator model it is necessary to have start and stop thresholds. In this paper we will develop a new model that varies response rate in relation to the likelihood of event occurrence, as opposed to a threshold, for changing the response rate. The new model reproduced the start and stop statistics that have been observed in 14 different PI experiments from 3 different papers. The developed model is also compared to the Time-adaptive Drift-diffusion Model (TDDM), the latest accumulator model subsuming the scalar expectancy theory (SET), on all 14 data-sets. The results show that it is unnecessary to have explicit start and stop thresholds or an internal equivalent to break-run-break states to reproduce the individual trials statistics and population behaviour and get the same break-run-break analysis results. The new model also produces more realistic individual trials compared to TDDM.

Animals can quickly learn the timing of events with fixed intervals and their rate of acquisition does not depend on the length of the interval. In contrast, recurrent neural networks that use gradient based learning have difficulty predicting the timing of events that depend on stimulus that occurred long ago. We present the latent time-adaptive drift-diffusion model (LTDDM), an extension to the time-adaptive drift-diffusion model (TDDM), a model for animal learning of timing that exhibits behavioural properties consistent with experimental data from animals. The performance of LTDDM is compared to that of a state of the art long short-term memory (LSTM) recurrent neural network across three timing tasks. Differences in the relative performance of these two models is discussed and it is shown how LTDDM can learn these events time series orders of magnitude faster than recurrent neural networks.

We aim to investigate the potential impacts of smart homes on human behavior. To this end, we simulate a series of human models capable of performing various activities inside a reinforcement learning-based smart home. We then investigate the possibility of human behavior being altered as a result of the smart home and the human model adapting to one-another. We design a semi-Markov decision process human task interleaving model based on hierarchical reinforcement learning that learns to make decisions to either pursue or leave an activity. We then integrate our human model in the smart home which is based on Q-learning. We show that a smart home trained on a generic human model is able to anticipate and learn the thermal preferences of human models with intrinsic rewards similar to the generic model. The hierarchical human model learns to complete each activity and set optimal thermal settings for maximum comfort. With the smart home, the number of time steps required to change the thermal settings are reduced for the human models. Interestingly, we observe that small variations in the human model reward structures can lead to the opposite behavior in the form of unexpected switching between activities which signals changes in human behavior due to the presence of the smart home., Comment: in IEEE Transactions on Artificial Intelligence

Animals learn the timing between consecutive events very easily. Their precision is usually proportional to the interval to time (Weber's law for timing). Most current timing models either require a central clock and unbounded accumulator or whole pre-defined populations of delay lines, decaying traces or oscillators to represent elapsing time. Current adaptive recurrent neural networks fail at learning to predict the timing of future events (the 'when') in a realistic manner. In this paper, we present a new model of interval timing, based on simple temporal integrators, derived from drift-diffusion models. We develop a simple geometric rule to learn 'when' instead of 'what'. We provide an analytical proof that the model can learn inter-event intervals in a number of trials independent of the interval size and that the temporal precision of the system is proportional to the timed interval. This new model uses no clock, no gradient, no unbounded accumulators, no delay lines, and has internal noise allowing generations of individual trials. Three interesting predictions are made., Comment: 9 pages, 4 figures