Search

In this paper we study the validity and efficiency of a conformal version of the CUSUM procedure for change detection both experimentally and theoretically., Comment: 19 pages, 7 figures

The usual way of testing probability forecasts in game-theoretic probability is via construction of test martingales. The standard assumption is that all forecasts are output by the same forecaster. In this paper I will discuss possible extensions of this picture to testing probability forecasts output by several forecasters. This corresponds to multiple hypothesis testing in statistics. One interesting phenomenon is that even a slight relaxation of the requirement of family-wise validity leads to a very significant increase in the efficiency of testing procedures. The main goal of this paper is to report results of preliminary simulation studies and list some directions of further research., Comment: 19 pages, 6 figures

This paper places conformal testing in a general framework of statistical hypothesis testing. A standard approach to testing a composite null hypothesis $H$ is to test each of its elements and to reject $H$ when each of its elements is rejected. It turns out that we can fully cover conformal testing using this approach only if we allow forgetting some of the data. However, we will see that the standard approach covers conformal testing in a weak asymptotic sense and under restrictive assumptions. I will also list several possible directions of further research, including developing a general scheme of online testing., Comment: 22 pages, 1 figure

A very simple example demonstrates that Fisher's application of the conditionality principle to regression ("fixed $x$ regression"), endorsed by Sprott and many other followers, makes prediction impossible in the context of statistical learning theory. On the other hand, relaxing the requirement of conditionality makes it possible via, e.g., conformal prediction., Comment: 6 pages

This note states a simple property of optimality of the Bayes-Kelly algorithm for conformal testing and poses a related open problem., Comment: 10 pages

This paper establishes the asymptotic uniqueness of long-term probability forecasts in the following form. Consider two forecasters who repeatedly issue probability forecasts for the infinite future. The main result of the paper says that either at least one of the two forecasters will be discredited or their forecasts will converge in total variation. This can be regarded as a game-theoretic version of the classical Blackwell-Dubins result getting rid of some of its limitations. This result is further strengthened along the lines of Richard Jeffrey's radical probabilism., Comment: 15 pages

Beginning in the 1970s, Alexander Philip Dawid has been a leading contributor to the foundations of statistics and especially to the development and application of Bayesian statistics. He is also known for his work on causality, especially his notation for conditional independence and his critique of the overuse of counterfactuals, and for his contributions to forensic statistics. Dawid was born in Lancashire, England, on February 1, 1946. His family moved to London soon afterwards, and he attended the City of London School from 1956 to 1963. He studied mathematics at Cambridge, earning a BA (Bachelor of Arts) degree in 1966. After earning a Diploma in Mathematical Statistics in the academic year 1966-1967, he studied for a PhD at Imperial, then at UCL, where he became a Lecturer in Statistics in 1969. In 1978, he left UCL for a position as Professor of Statistics in the Department of Mathematics, The City University, London, where he served as Head of Statistics Section and Director of the Statistical Laboratory. He returned to the Department of Statistics at UCL in 1981, serving as Head of Department from 1983 to 1993. He moved to the University of Cambridge in 2007, becoming Professor of Statistics and Fellow of Darwin College. He has continued his work in mathematical statistics after retiring from Cambridge in 2013 and was elected Fellow of the Royal Society in 2018., Comment: 46 pages, 4 figures; short version is to appear in "Statistical Science"

In this paper I discuss both syntax and semantics of subjective probability. The semantics determines ways of testing probability statements. Among important varieties of subjective probabilities are intersubjective probabilities and impersonal probabilities, and I will argue that well-tested impersonal probabilities acquire features of objective probabilities. Jeffreys's law, my next topic, states that two successful probability forecasters must issue forecasts that are close to each other, thus supporting the idea of objective probabilities. Finally, I will discuss connections between subjective and frequentist probability., Comment: 19 pages

It is well known that a Bayesian probability forecast for all future observations should be a probability measure in order to satisfy a natural condition of coherence. The main topics of this paper are the evolution of the Bayesian probability measure and ways of testing its adequacy as it evolves over time. The process of testing evolving Bayesian beliefs is modelled in terms of betting, similarly to the standard Dutch book treatment of coherence. The resulting picture is adapted to forecasting several steps ahead and making almost optimal decisions., Comment: 44 pages; old results improved, new results and discussions added

The topic of this paper is testing exchangeability using e-values in the batch mode, with the Markov model as alternative. The null hypothesis of exchangeability is formalized as a Kolmogorov-type compression model, and the Bayes mixture of the Markov model w.r. to the uniform prior is taken as simple alternative hypothesis. Using e-values instead of p-values leads to a computationally efficient testing procedure. In the appendixes I explain connections with the algorithmic theory of randomness and with the traditional theory of testing statistical hypotheses. In the standard statistical terminology, this paper proposes a new permutation test. This test can also be interpreted as a poor man's version of Kolmogorov's deficiency of randomness., Comment: 10 pages, 2 figures, 3 tables

Safe anytime-valid inference (SAVI) provides measures of statistical evidence and certainty -- e-processes for testing and confidence sequences for estimation -- that remain valid at all stopping times, accommodating continuous monitoring and analysis of accumulating data and optional stopping or continuation for any reason. These measures crucially rely on test martingales, which are nonnegative martingales starting at one. Since a test martingale is the wealth process of a player in a betting game, SAVI centrally employs game-theoretic intuition, language and mathematics. We summarize the SAVI goals and philosophy, and report recent advances in testing composite hypotheses and estimating functionals in nonparametric settings., Comment: 25 pages. Under review. ArXiv does not compile/space some references properly

The notion of an e-value has been recently proposed as a possible alternative to critical regions and p-values in statistical hypothesis testing. In this paper we consider testing the nonparametric hypothesis of symmetry, introduce analogues for e-values of three popular nonparametric tests, define an analogue for e-values of Pitman's asymptotic relative efficiency, and apply it to the three nonparametric tests. We discuss limitations of our simple definition of asymptotic relative efficiency and list directions of further research., Comment: 21 pages, 4 figures, 1 table

We continue study of conformal testing in binary model situations. In this note we consider Markov alternatives to the null hypothesis of exchangeability. We propose two new classes of conformal test martingales; one class is statistically efficient in our experiments, and the other class partially sacrifices statistical efficiency to gain computational efficiency., Comment: 8 pages, 8 figures

This paper proposes a way of protecting probabilistic prediction models against changes in the data distribution, concentrating on the case of classification and paying particular attention to binary classification. This is important in applications of machine learning, where the quality of a trained prediction algorithm may drop significantly in the process of its exploitation. Our techniques are based on recent work on conformal test martingales and older work on prediction with expert advice, namely tracking the best expert., Comment: 23 pages, 14 figures, and 4 tables

This note proposes a procedure for enhancing the quality of probabilistic prediction algorithms via betting against their predictions. It is inspired by the success of the conformal test martingales that have been developed recently., Comment: 7 pages, 4 figures

Conformal testing is a way of testing the IID assumption based on conformal prediction. The topic of this note is computational evaluation of the performance of conformal testing in a model situation in which IID binary observations generated from a Bernoulli distribution are followed by IID binary observations generated from another Bernoulli distribution, with the parameters of the distributions and changepoint unknown. Existing conformal test martingales can be used for this task and work well in simple cases, but their efficiency can be improved greatly., Comment: 8 pages, 5 figures

We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency., Comment: 22 pages, 19 figures, 3 tables

This note continues study of exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the observations. Such processes can be used for detecting violations of the IID assumption, which is commonly made in machine learning. Violations of the IID assumption are sometimes referred to as dataset shift, and dataset shift is sometimes subdivided into concept shift, covariate shift, etc. Our primary interest is in concept shift, but we will also discuss exchangeability martingales that decompose perfectly into two components one of which detects concept shift and the other detects what we call label shift. Our methods will be based on techniques of conformal prediction., Comment: 14 pages, 2 figures

We explain, on the example of Wigner's quasiprobability distribution, how negative probabilities may be used in the foundations of probability., Comment: 6 pages

This note is my comment on Glenn Shafer's discussion paper "Testing by betting", together with two online appendices comparing p-values and betting scores., Comment: 7 pages, 1 figure

This note reanalyzes Cox's idealized example of testing with data splitting using e-values (Shafer's betting scores). Cox's exciting finding was that the method of data splitting, while allowing flexible data analysis, achieves quite high efficiencies, of about 80%. The most serious objection to the method was that it involves splitting data at random, and so different people analyzing the same data may get very different answers. Using e-values instead of p-values remedies this disadvantage., Comment: 11 pages, 8 figures, 2 tables

Methods of merging several p-values into a single p-value are important in their own right and widely used in multiple hypothesis testing. This paper is the first to systematically study the admissibility (in Wald's sense) of p-merging functions and their domination structure, without any information on the dependence structure of the input p-values. As a technical tool we use the notion of e-values, which are alternatives to p-values recently promoted by several authors. We obtain several results on the representation of admissible p-merging functions via e-values and on (in)admissibility of existing p-merging functions. By introducing new admissible p-merging functions, we show that some classic merging methods can be strictly improved to enhance power without compromising validity under arbitrary dependence., Comment: 43 pages and 9 figures; as compared with the previous version, there are numerous improvements and further simulation studies

We study the problem of merging sequential or independent e-values into one e-value or e-process. We describe a class of e-value merging functions via martingales and show that it dominates all merging methods for sequential e-values. All admissible methods for constructing e-processes can also be obtained in this way. In the case of merging independent e-values, the situation becomes much more complicated, and we provide a general class of such merging functions based on martingales applied to reordered data., Comment: 21 pages

There is a useful counterpart of conformal prediction for e-values, called conformal e-prediction. Conformal prediction can serve as basis for testing the assumption of exchangeability, leading to conformal testing. Similarly, conformal e-prediction can also serve as basis for testing. The resulting conformal e-testing looks very different from but inherits some strengths of conformal testing; it even has some advantages over conformal testing. In this paper we discuss systematically both strengths and limitations of conformal e-testing., Comment: 21 pages and 2 figures

Efficiency criteria for conformal prediction, such as \emph{observed fuzziness} (i.e., the sum of p-values associated with false labels), are commonly used to \emph{evaluate} the performance of given conformal predictors. Here, we investigate whether it is possible to exploit efficiency criteria to \emph{learn} classifiers, both conformal predictors and point classifiers, by using such criteria as training objective functions. The proposed idea is implemented for the problem of binary classification of hand-written digits. By choosing a 1-dimensional model class (with one real-valued free parameter), we can solve the optimization problems through an (approximate) exhaustive search over (a discrete version of) the parameter space. Our empirical results suggest that conformal predictors trained by minimizing their observed fuzziness perform better than conformal predictors trained in the traditional way by minimizing the \emph{prediction error} of the corresponding point classifier. They also have a reasonable performance in terms of their prediction error on the test set., Comment: 8 pages, 2 figures, 4 tables

In this paper we use e-values in the context of multiple hypothesis testing assuming that the base tests produce independent, or sequential, e-values. Our simulation and empirical studies and theoretical considerations suggest that, under this assumption, our new algorithms are superior to the known algorithms using independent p-values and to our recent algorithms designed for e-values without the assumption of independence., Comment: 24 pages, 5 figures

This paper discusses a counterpart of conformal prediction for e-values, conformal e-prediction. Conformal e-prediction is conceptually simpler and had been developed in the 1990s as precursor of conformal prediction. When conformal prediction emerged as result of replacing e-values by p-values, it seemed to have important advantages over conformal e-prediction without obvious disadvantages. This paper re-examines relations between conformal prediction and conformal e-prediction systematically from a modern perspective. Conformal e-prediction has advantages of its own, such as the ease of designing conditional conformal e-predictors and the guaranteed validity of cross-conformal e-predictors (whereas for cross-conformal predictors validity is only an empirical fact and can be broken with excessive randomization). Even where conformal prediction has clear advantages, conformal e-prediction can often emulate those advantages, more or less successfully., Comment: The previous version has been split into conformal e-prediction (this arXiv report) and conformal e-testing (arXiv report 2006.02329); 29 pages and 1 figure

We discuss systematically two versions of confidence regions: those based on p-values and those based on e-values, a recent alternative to p-values. Both versions can be applied to multiple hypothesis testing, and in this paper we are interested in procedures that control the number of false discoveries under arbitrary dependence between the base p- or e-values. We introduce a procedure that is based on e-values and show that it is efficient both computationally and statistically using simulated and real-world datasets. Comparison with the corresponding standard procedure based on p-values is not straightforward, but there are indications that the new one performs significantly better in some situations., Comment: 50 pages, 13 figures, and 5 tables

Multiple testing of a single hypothesis and testing multiple hypotheses are usually done in terms of p-values. In this paper we replace p-values with their natural competitor, e-values, which are closely related to betting, Bayes factors, and likelihood ratios. We demonstrate that e-values are often mathematically more tractable; in particular, in multiple testing of a single hypothesis, e-values can be merged simply by averaging them. This allows us to develop efficient procedures using e-values for testing multiple hypotheses., Comment: 48 pages, 5 figures, 4 algorithms. A new title and improved presentation

Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conformal predictive systems inherit the relative computational inefficiency of conformal predictors. In this paper we discuss two computationally efficient versions of conformal predictive systems, which we call split conformal predictive systems and cross-conformal predictive systems. The main advantage of split conformal predictive systems is their guaranteed validity, whereas for cross-conformal predictive systems validity only holds empirically and in the absence of excessive randomization. The main advantage of cross-conformal predictive systems is their greater predictive efficiency., Comment: 31 pages, 14 figures, 1 table. The conference version published in the Proceedings of COPA 2018, and the journal version is to appear in Neurocomputing

This paper proposes an alternative language for expressing results of the algorithmic theory of randomness. The language is more precise in that it does not involve unspecified additive or multiplicative constants, making mathematical results, in principle, applicable in practice. Our main testing ground for the proposed language is the problem of defining Bernoulli sequences, which was of great interest to Andrei Kolmogorov and his students., Comment: 15 pages

The hypothesis of randomness is fundamental in statistical machine learning and in many areas of nonparametric statistics; it says that the observations are assumed to be independent and coming from the same unknown probability distribution. This hypothesis is close, in certain respects, to the hypothesis of exchangeability, which postulates that the distribution of the observations is invariant with respect to their permutations. This paper reviews known methods of testing the two hypotheses concentrating on the online mode of testing, when the observations arrive sequentially. All known online methods for testing these hypotheses are based on conformal martingales, which are defined and studied in detail. The paper emphasizes conceptual and practical aspects and states two kinds of results. Validity results limit the probability of a false alarm or the frequency of false alarms for various procedures based on conformal martingales, including conformal versions of the CUSUM and Shiryaev-Roberts procedures. Efficiency results establish connections between randomness, exchangeability, and conformal martingales., Comment: 34 pages, 1 table, 4 figures

Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems that are fully adaptive. Our method consists in calibrating existing predictive systems; the input predictive system is not supposed to satisfy any properties of validity, whereas the output predictive system is guaranteed to be calibrated in probability. It is interesting that the method may also work without the IID assumption, standard in conformal prediction., Comment: 10 pages, 2 figures

April 25, 2003, marked the 100th anniversary of the birth of Andrei Nikolaevich Kolmogorov, the twentieth century's foremost contributor to the mathematical and philosophical foundations of probability. The year 2003 was also the 70th anniversary of the publication of Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung. Kolmogorov's Grundbegriffe put probability's modern mathematical formalism in place. It also provided a philosophy of probability - an explanation of how the formalism can be connected to the world of experience. In this article, we examine the sources of these two aspects of the Grundbegriffe - the work of the earlier scholars whose ideas Kolmogorov synthesized., Comment: 104 pages. arXiv admin note: substantial text overlap with arXiv:math/0606533

We derive formulas for the performance of capital assets in continuous time from an efficient market hypothesis, with no stochastic assumptions and no assumptions about the beliefs or preferences of investors. Our efficient market hypothesis says that a speculator with limited means cannot beat a particular index by a substantial factor. Our results include a formula that resembles the classical CAPM formula for the expected simple return of a security or portfolio. This version of the article was essentially written in December 2001 but remains a working paper., Comment: 10 pages

We study the origins of the $\sqrt{dt}$ effect in finance and SDE. In particular, we show, in the game-theoretic framework, that market volatility is a consequence of the absence of riskless opportunities for making money and that too high volatility is also incompatible with such opportunities. More precisely, riskless opportunities for making money arise whenever a traded security has fractal dimension below or above that of the Brownian motion and its price is not almost constant and does not become extremely large. This is a simple observation known in the measure-theoretic mathematical finance. At the end of the article we also consider the case of non-zero interest rate. This version of the article was essentially written in March 2005 but remains a working paper., Comment: 17 pages