Search

This work introduces a novel, simple, and flexible method to quantify irreversibility in generic high-dimensional time series based on the well-known mapping to a binary classification problem. Our approach utilizes gradient boosting for estimation, providing a model-free, nonlinear analysis able to handle large-dimensional systems while requiring minimal or no calibration. Our procedure is divided into three phases: trajectory encoding, Markovian order identification, and hypothesis testing for variable interactions. The latter is the key innovation that allows us to selectively switch off variable interactions to discern their specific contribution to irreversibility. When applied to financial markets, our findings reveal a distinctive shift: during stable periods, irreversibility is mainly related to short-term patterns, whereas in unstable periods, these short-term patterns are disrupted, leaving only contributions from stable long-term ones. This observed transition underscores the crucial importance of high-order variable interactions in understanding the dynamics of financial markets, especially in times of turbulence.

Obtaining an accurate estimate of the underlying covariance matrix from finite sample size data is challenging due to sample size noise. In recent years, sophisticated covariance-cleaning techniques based on random matrix theory have been proposed to address this issue. Most of these methods aim to achieve an optimal covariance matrix estimator by minimizing the Frobenius norm distance as a measure of the discrepancy between the true covariance matrix and the estimator. However, this practice offers limited interpretability in terms of information theory. To better understand this relationship, we focus on the Kullback-Leibler divergence to quantify the information lost by the estimator. Our analysis centers on rotationally invariant estimators, which are state-of-art in random matrix theory, and we derive an analytical expression for their Kullback-Leibler divergence. Due to the intricate nature of the calculations, we use genetic programming regressors paired with human intuition. Ultimately, using this approach, we formulate a conjecture validated through extensive simulations, showing that the Frobenius distance corresponds to a first-order expansion term of the Kullback-Leibler divergence, thus establishing a more defined link between the two measures.

The Average Oracle, a simple and very fast covariance filtering method, is shown to yield superior Sharpe ratios than the current state-of-the-art (and complex) methods, Dynamic Conditional Covariance coupled to Non-Linear Shrinkage (DCC+NLS). We pit all the known variants of DCC+NLS (quadratic shrinkage, gross-leverage or turnover limitations, and factor-augmented NLS) against the Average Oracle in large-scale randomized experiments. We find generically that while some variants of DCC+NLS sometimes yield the lowest average realized volatility, albeit with a small improvement, their excessive gross leverage and investment concentration, and their 10-time larger turnover contribute to smaller average portfolio returns, which mechanically result in smaller realized Sharpe ratios than the Average Oracle. We also provide simple analytical arguments about the origin of the advantage of the Average Oracle over NLS in a changing world., Comment: 9 pages, 2 figures, 6 tables

In optimal covariance cleaning theory, minimizing the Frobenius norm between the true population covariance matrix and a rotational invariant estimator is a key step. This estimator can be obtained asymptotically for large covariance matrices, without knowledge of the true covariance matrix. In this study, we demonstrate that this minimization problem is equivalent to minimizing the loss of information between the true population covariance and the rotational invariant estimator for normal multivariate variables. However, for Student's t distributions, the minimal Frobenius norm does not necessarily minimize the information loss in finite-sized matrices. Nevertheless, such deviations vanish in the asymptotic regime of large matrices, which might extend the applicability of random matrix theory results to Student's t distributions. These distributions are characterized by heavy tails and are frequently encountered in real-world applications such as finance, turbulence, or nuclear physics. Therefore, our work establishes a connection between statistical random matrix theory and estimation theory in physics, which is predominantly based on information theory.

Symbolic transfer entropy is a powerful non-parametric tool to detect lead-lag between time series. Because a closed expression of the distribution of Transfer Entropy is not known for finite-size samples, statistical testing is often performed with bootstraps whose slowness prevents the inference of large lead-lag networks between long time series. On the other hand, the asymptotic distribution of Transfer Entropy between two time series is known. In this work, we derive the asymptotic distribution of the test for one time series having a larger Transfer Entropy than another one on a target time series. We then measure the convergence speed of both tests in the small sample size limits via benchmarks. We then introduce Transfer Entropy between time-shifted time series, which allows to measure the timescale at which information transfer is maximal and vanishes. We finally apply these methods to tick-by-tick price changes of several hundreds of stocks, yielding non-trivial statistically validated networks.

Portfolio optimization requires sophisticated covariance estimators that are able to filter out estimation noise. Non-linear shrinkage is a popular estimator based on how the Oracle eigenvalues can be computed using only data from the calibration window. Contrary to common belief, NLS is not optimal for portfolio optimization because it does not minimize the right cost function when the asset dependence structure is non-stationary. We instead derive the optimal target. Using historical data, we quantify by how much NLS can be improved. Our findings reopen the question of how to build the covariance matrix estimator for portfolio optimization in realistic conditions.

We propose a data-driven way to reduce the noise of covariance matrices of nonstationary systems. In the case of stationary systems, asymptotic approaches were proved to converge to the optimal solutions. Such methods produce eigenvalues that are highly dependent on the inputs, as common sense would suggest. Our approach proposes instead to use a set of eigenvalues totally independent from the inputs and that encode the long-term averaging of the influence of the future on present eigenvalues. Such an influence can be the predominant factor in nonstationary systems. Using real and synthetic data, we show that our data-driven method outperforms optimal methods designed for stationary systems for the filtering of both covariance matrix and its inverse, as illustrated by financial portfolio variance minimization, which makes out method generically relevant to many problems of multivariate inference.

How do pedestrians choose their paths within city street networks? Researchers have tried to shed light on this matter through strictly controlled experiments, but an ultimate answer based on real-world mobility data is still lacking. Here, we analyze salient features of human path planning through a statistical analysis of a massive dataset of GPS traces, which reveals that (1) people increasingly deviate from the shortest path when the distance between origin and destination increases, and (2) chosen paths are statistically different when origin and destination are swapped. We posit that direction to goal is a main driver of path planning and develop a vector-based navigation model that is a statistically better predictor of human paths than a model based on minimizing distance with stochastic effects. Our findings generalize across two major US cities with different street networks, hinting to the fact that vector-based navigation might be a universal property of human path planning.

We apply the knockoff procedure to factor selection in finance. By building fake but realistic factors, this procedure makes it possible to control the fraction of false discovery in a given set of factors. To show its versatility, we apply it to fund replication and to the inference of explanatory and prediction networks.

We propose a multi-layer network model for the spread of COVID-19 that accounts for interactions within the family, between schoolmates, and casual contacts in the population. We utilize the proposed model-calibrated on epidemiological and demographic data-to investigate current questions concerning the implementation of non-pharmaceutical interventions (NPIs) during the vaccination campaign. Specifically, we consider scenarios in which the most fragile population has already received the vaccine, and we focus our analysis on the role of schools as drivers of the contagions and on the implementation of targeted intervention policies oriented to children and their families. We perform our analysis by means of a campaign of Monte Carlo simulations. Our findings suggest that, in a phase with NPIs enacted but in-person education, children play a key role in the spreading of COVID-19. Interestingly, we show that children's testing might be an important tool to flatten the epidemic curve, in particular when combined with enacting temporary online education for classes in which infected students are detected. Finally, we test a vaccination strategy that prioritizes the members of large families and we demonstrate its good performance. We believe that our modeling framework and our findings could be of help for public health authorities for planning their current and future interventions, as well as to increase preparedness for future epidemic outbreaks.

We introduce a $k$-fold boosted version of our Boostrapped Average Hierarchical Clustering cleaning procedure for correlation and covariance matrices. We then apply this method to global minimum variance portfolios for various values of $k$ and compare their performance with other state-of-the-art methods. Generally, we find that our method yields better Sharpe ratios after transaction costs than competing filtering methods, despite requiring a larger turnover.

I show analytically that the average of $k$ bootstrapped correlation matrices rapidly becomes positive-definite as $k$ increases, which provides a simple approach to regularize singular Pearson correlation matrices. If $n$ is the number of objects and $t$ the number of features, the averaged correlation matrix is almost surely positive-definite if $k> \frac{e}{e-1}\frac{n}{t}\simeq 1.58\frac{n}{t}$ in the limit of large $t$ and $n$. The probability of obtaining a positive-definite correlation matrix with $k$ bootstraps is also derived for finite $n$ and $t$. Finally, I demonstrate that the number of required bootstraps is always smaller than $n$. This method is particularly relevant in fields where $n$ is orders of magnitude larger than the size of data points $t$, e.g., in finance, genetics, social science, or image processing.

Statistical inference of the dependence between objects often relies on covariance matrices. Unless the number of features (e.g. data points) is much larger than the number of objects, covariance matrix cleaning is necessary to reduce estimation noise. We propose a method that is robust yet flexible enough to account for fine details of the structure covariance matrix. Robustness comes from using a hierarchical ansatz and dependence averaging between clusters; flexibility comes from a bootstrap procedure. This method finds several possible hierarchical structures in DNA microarray gene expression data, and leads to lower realized risk in global minimum variance portfolios than current filtering methods when the number of data points is relatively small., Comment: 10 pages, 9 figures

We introduce a method to predict which correlation matrix coefficients are likely to change their signs in the future in the high-dimensional regime, i.e. when the number of features is larger than the number of samples per feature. The stability of correlation signs, two-by-two relationships, is found to depend on three-by-three relationships inspired by Heider social cohesion theory in this regime. We apply our method to US and Hong Kong equities historical data to illustrate how the structure of correlation matrices influences the stability of the sign of its coefficients.

We develop a greedy algorithm that is fast and scalable in the detection of a nested partition extracted from a dendrogram obtained from hierarchical clustering of a multivariate series. Our algorithm provides a $p$-value for each clade observed in the hierarchical tree. The $p$-value is obtained by computing a number of bootstrap replicas of the dissimilarity matrix and by performing a statistical test on each difference between the dissimilarity associated with a given clade and the dissimilarity of the clade of its parent node. We prove the efficacy of our algorithm with a set of benchmarks generated by using a hierarchical factor model. We compare the results obtained by our algorithm with those of Pvclust. Pvclust is a widely used algorithm developed with a global approach originally motivated by phylogenetic studies. In our numerical experiments we focus on the role of multiple hypothesis test correction and on the robustness of the algorithms to inaccuracy and errors of datasets. We also apply our algorithm to a reference empirical dataset. We verify that our algorithm is much faster than Pvclust algorithm and has a better scalability both in the number of elements and in the number of records of the investigated multivariate set. Our algorithm provides a hierarchically nested partition in much shorter time than currently widely used algorithms allowing to perform a statistically validated cluster analysis detection in very large systems.

We use the information present in a bipartite network to detect cores of communities of each set of the bipartite system. Cores of communities are found by investigating statistically validated projected networks obtained using information present in the bipartite network. Cores of communities are highly informative and robust with respect to the presence of errors or missing entries in the bipartite network. We assess the statistical robustness of cores by investigating an artificial benchmark network, the co-authorship network, and the actor-movie network. The accuracy and precision of the partition obtained with respect to the reference partition are measured in terms of the adjusted Rand index and of the adjusted Wallace index respectively. The detection of cores is highly precise although the accuracy of the methodology can be limited in some cases., Comment: 9 pages, 6 figures

In this paper we present a simulator allowing to perform policy experiments relative to the air traffic management. Different SESAR solutions can be implemented in the model to see the reaction of the different stakeholders as well as other relevant metrics (delays, safety, etc). The model describes both the strategic phase associated to the planning of the flight trajectories and the tactical modifications occurring in the en-route phase. An implementation of the model is available as open-source and freely accessible by any user. More specifically, different procedures related to business trajectories and free-routing are tested and we illustrate the capabilities of the model on airspace implementing these concepts. After performing numerical simulations with the model, we show that in a free-routing scenario the controllers perform less operations although they are dispersed over a larger portion of the airspace. This can potentially increase the complexity of conflict detection and resolution for controllers. In order to investigate this specific aspect, we consider some metrics used to measure traffic complexity. We first show that in non-free-routing situations our simulator deals with complexity in a way similar to what humans would do. This allows us to be confident that the results of our numerical simulations relative to the free-routing can reasonably forecast how human controllers would behave in this new situation. Specifically, our numerical simulations show that most of the complexity metrics decrease with free-routing, while the few metrics which increase are all linked to the flight level changes. This is a non-trivial result since intuitively the complexity should increase with free-routing because of problematic geometries and more dispersed conflicts over the airspace.