Search

Algorithmic complexity provides a mathematical formal notion of string complexity. Building on this, one arrives at mathematical ‘gold standard’ (though incomputable) definitions of randomness, induction, similarity, and even intelligence. These definitions can be turned into practical algorithms by using common compressors to approximate the universal solutions. One can consider the theories as idealized cognition with respect to which one can aim to describe actual biological cognition by listing biases and limitations that need to be defined relative to some normative reference.

Feature reinforcement learning was introduced five years ago as a principled and practical approach to history-based learning. This paper examines the progress since its inception. We now have both model-based and model-free cost functions, most recently extended to the function approximation setting. Our current work is geared towards playing ATARI games using imitation learning, where we use Feature RL as a feature selection method for high-dimensional domains.

The problem we consider in this paper is reinforcement learning with value advice. In this setting, the agent is given limited access to an oracle that can tell it the expected return (value) of any state-action pair with respect to the optimal policy. The agent must use this value to learn an explicit policy that performs well in the environment. We provide an algorithm called RLAdvice, based on the imitation learning algorithm DAgger. We illustrate the effectiveness of this method in the Arcade Learning Environment on three different games, using value estimates from UCT as advice.

We extend the Q-learning algorithm from the Markov Decision Process setting to problems where observations are non-Markov and do not reveal the full state of the world i.e. to POMDPs. We do this in a natural manner by adding l0 regularisation to the pathwise squared Q-learning objective function and then optimise this over both a choice of map from history to states and the resulting MDP parameters. The optimisation procedure involves a stochastic search over the map class nested with classical Q-learning of the parameters. This algorithm fits perfectly into the feature reinforcement learning framework, which chooses maps based on a cost criteria. The cost criterion used so far for feature reinforcement learning has been model-based and aimed at predicting future states and rewards. Instead we directly predict the return, which is what is needed for choosing optimal actions. Our Q-learning criteria also lends itself immediately to a function approximation setting where features are chosen based on the history. This algorithm is somewhat similar to the recent line of work on lasso temporal difference learning which aims at finding a small feature set with which one can perform policy evaluation. The distinction is that we aim directly for learning the Q-function of the optimal policy and we use l0 instead of l1 regularisation. We perform an experimental evaluation on classical benchmark domains and find improvement in convergence speed as well as in economy of the state representation. We also compare against MC-AIXI on the large Pocman domain and achieve competitive performance in average reward. We use less than half the CPU time and 36 times less memory. Overall, our algorithm hQL provides a better combination of computational, memory and data efficiency than existing algorithms in this setting.

Online adaptation of Brain Computer Interfaces allows for arduous training periods to be circumvented. To do this we must adapt a classifier to a new session, or better yet, a new subject. We initially outline a procedure to perform online adaptation of b

An interesting scheme for estimating and adapting distributions in real-time for non-stationary data has recently been the focus of study for several different tasks relating to time series and data mining, namely change point detection, outlier detection and online compression/sequence prediction. An appealing feature is that unlike more sophisticated procedures, it is as fast as the related stationary procedures which are simply modified through discounting or windowing. The discount scheme makes older observations lose their influence on new predictions. The authors of this article recently used a discount scheme for introducing an adaptive version of the Context Tree Weighting compression algorithm. The mentioned change point and outlier detection methods rely on the changing compression ratio of an online compression algorithm. Here we are beginning to provide theoretical foundations for the use of these adaptive estimation procedures that have already shown practical promise.

Automated channel selection allows the dimension of EEG data to be reduced without expert knowledge. We introduce Recursive Channel Insertion, an extension to Recursive Channel Elimination, which dramatically reduces calculation time with no loss of accur

Ideal Brain Computer Interfaces need to perform asynchronously and at real time. We propose Hidden Semi-Markov Models(HSMM) to better segment and classify EEG data. The proposed HSMM method was tested against a simple windowed method on standard datasets. We found that our HSMM outperformed the simple windowed method. Furthermore, due to the computational demands of the algorithm, we adapted the HSMM algorithm to an online setting and demonstrate that this faster version of the algorithm can run in real time.

We describe an adaptive context tree weighting (ACTW) algorithm, as an extension to the standard context tree weighting (CTW) algorithm. Unlike the standard CTW algorithm, which weights all observations equally regardless of the depth, ACTW gives increasing weight to more recent observations, aiming to improve performance in cases where the input sequence is from a non-stationary distribution. Data compression results show ACTW variants improving over CTW on merged files from standard compression benchmark tests while never being significantly worse on any individual file.

We introduce the first online kernelized version of SARSA(?) to permit sparsifi- cation for arbitrary ? for 0 = ? = 1; this is possible via a novel kernelization of the eligibility trace that is maintained separately from the kernelized value function. This separation is crucial for preserving the functional structure of the eligibility trace when using sparse kernel projection techniques that are essential for memory efficiency and capacity control. The result is a simple and practical Kernel-SARSA(?) algorithm for general 0 = ? = 1 that is memory-efficient in comparison to standard SARSA(?) (using various basis functions) on a range of domains including a real robotics task running on a Willow Garage PR2 robot.

Detailed monitoring of training sessions of elite athletes is an important component of their training. In this paper we describe an application that performs a precise segmentation and labeling of swimming sessions. This allows a comprehensive breakdown

Subsequence clustering aims to find patterns that appear repeatedly in time series data. We introduce a novel subsequence clustering technique that we call semi-Markov kmeans clustering. The clustering results in ideal examples of the repeating patterns and in labeled segmentations that can be used as training data for sophisticated discriminative methods like max-margin semi-Markov models. We are applying the new clustering technique to activity recognition from body-worn sensors by showing how it can enable a system to learn from data that is only annotated by an ordered list of activity types that have been undertaken. This kind of annotation, unlike a detailed segmentation of the sensor data, is easily provided by a non-expert user. We show that we can achieve equally good results using only an ordered list of activity types for training as when using a full detailed labeled segmentation.

The Hammersley-Clifford (H-C) theorem relates the factorization properties of a probability distribution to the clique structure of an undirected graph. If a density factorizes according to the clique structure of an undirected graph, the theorem guarantees that the distribution satisfies the Markov property and vice versa. We show how to generalize the H-C theorem to different notions of decomposability and the corresponding generalized-Markov property. Finally we discuss how our technique might be used to arrive at other generalizations of the H-C theorem, inducing a graph semantics adapted to the modeling problem.