Search

In optimal covariance cleaning theory, minimizing the Frobenius norm between the true population covariance matrix and a rotational invariant estimator is a key step. This estimator can be obtained asymptotically for large covariance matrices, without knowledge of the true covariance matrix. In this study, we demonstrate that this minimization problem is equivalent to minimizing the loss of information between the true population covariance and the rotational invariant estimator for normal multivariate variables. However, for Student's t distributions, the minimal Frobenius norm does not necessarily minimize the information loss in finite-sized matrices. Nevertheless, such deviations vanish in the asymptotic regime of large matrices, which might extend the applicability of random matrix theory results to Student's t distributions. These distributions are characterized by heavy tails and are frequently encountered in real-world applications such as finance, turbulence, or nuclear physics. Therefore, our work establishes a connection between statistical random matrix theory and estimation theory in physics, which is predominantly based on information theory.

Symbolic transfer entropy is a powerful non-parametric tool to detect lead-lag between time series. Because a closed expression of the distribution of Transfer Entropy is not known for finite-size samples, statistical testing is often performed with bootstraps whose slowness prevents the inference of large lead-lag networks between long time series. On the other hand, the asymptotic distribution of Transfer Entropy between two time series is known. In this work, we derive the asymptotic distribution of the test for one time series having a larger Transfer Entropy than another one on a target time series. We then measure the convergence speed of both tests in the small sample size limits via benchmarks. We then introduce Transfer Entropy between time-shifted time series, which allows to measure the timescale at which information transfer is maximal and vanishes. We finally apply these methods to tick-by-tick price changes of several hundreds of stocks, yielding non-trivial statistically validated networks.

A common misconception is that the Oracle eigenvalue estimator of the covariance matrix yields the best realized portfolio performance. In reality, the Oracle estimator simply modifies the empirical covariance matrix eigenvalues so as to minimize the Frobenius distance between the filtered and the realized covariance matrices. This leads to the best portfolios only when the in-sample eigenvectors coincide with the out-of-sample ones. In all the other cases, the optimal eigenvalue correction can be obtained from the solution of a Quadratic-Programming problem. Solving it shows that the Oracle estimators only yield the best portfolios in the limit of infinite data points per asset and only in stationary systems.

We propose a data-driven way to reduce the noise of covariance matrices of nonstationary systems. In the case of stationary systems, asymptotic approaches were proved to converge to the optimal solutions. Such methods produce eigenvalues that are highly dependent on the inputs, as common sense would suggest. Our approach proposes instead to use a set of eigenvalues totally independent from the inputs and that encode the long-term averaging of the influence of the future on present eigenvalues. Such an influence can be the predominant factor in nonstationary systems. Using real and synthetic data, we show that our data-driven method outperforms optimal methods designed for stationary systems for the filtering of both covariance matrix and its inverse, as illustrated by financial portfolio variance minimization, which makes out method generically relevant to many problems of multivariate inference.

We develop a greedy algorithm that is fast and scalable in the detection of a nested partition extracted from a dendrogram obtained from hierarchical clustering of a multivariate series. Our algorithm provides a $p$-value for each clade observed in the hierarchical tree. The $p$-value is obtained by computing a number of bootstrap replicas of the dissimilarity matrix and by performing a statistical test on each difference between the dissimilarity associated with a given clade and the dissimilarity of the clade of its parent node. We prove the efficacy of our algorithm with a set of benchmarks generated by using a hierarchical factor model. We compare the results obtained by our algorithm with those of Pvclust. Pvclust is a widely used algorithm developed with a global approach originally motivated by phylogenetic studies. In our numerical experiments we focus on the role of multiple hypothesis test correction and on the robustness of the algorithms to inaccuracy and errors of datasets. We also apply our algorithm to a reference empirical dataset. We verify that our algorithm is much faster than Pvclust algorithm and has a better scalability both in the number of elements and in the number of records of the investigated multivariate set. Our algorithm provides a hierarchically nested partition in much shorter time than currently widely used algorithms allowing to perform a statistically validated cluster analysis detection in very large systems.

We propose a multi-layer network model for the spread of COVID-19 that accounts for interactions within the family, between schoolmates, and casual contacts in the population. We utilize the proposed model-calibrated on epidemiological and demographic data-to investigate current questions concerning the implementation of non-pharmaceutical interventions (NPIs) during the vaccination campaign. Specifically, we consider scenarios in which the most fragile population has already received the vaccine, and we focus our analysis on the role of schools as drivers of the contagions and on the implementation of targeted intervention policies oriented to children and their families. We perform our analysis by means of a campaign of Monte Carlo simulations. Our findings suggest that, in a phase with NPIs enacted but in-person education, children play a key role in the spreading of COVID-19. Interestingly, we show that children's testing might be an important tool to flatten the epidemic curve, in particular when combined with enacting temporary online education for classes in which infected students are detected. Finally, we test a vaccination strategy that prioritizes the members of large families and we demonstrate its good performance. We believe that our modeling framework and our findings could be of help for public health authorities for planning their current and future interventions, as well as to increase preparedness for future epidemic outbreaks.

We introduce a $k$-fold boosted version of our Boostrapped Average Hierarchical Clustering cleaning procedure for correlation and covariance matrices. We then apply this method to global minimum variance portfolios for various values of $k$ and compare their performance with other state-of-the-art methods. Generally, we find that our method yields better Sharpe ratios after transaction costs than competing filtering methods, despite requiring a larger turnover.

I show analytically that the average of $k$ bootstrapped correlation matrices rapidly becomes positive-definite as $k$ increases, which provides a simple approach to regularize singular Pearson correlation matrices. If $n$ is the number of objects and $t$ the number of features, the averaged correlation matrix is almost surely positive-definite if $k> \frac{e}{e-1}\frac{n}{t}\simeq 1.58\frac{n}{t}$ in the limit of large $t$ and $n$. The probability of obtaining a positive-definite correlation matrix with $k$ bootstraps is also derived for finite $n$ and $t$. Finally, I demonstrate that the number of required bootstraps is always smaller than $n$. This method is particularly relevant in fields where $n$ is orders of magnitude larger than the size of data points $t$, e.g., in finance, genetics, social science, or image processing.

The present work aims at proposing an analysis of cruise passengers trajectories at the destination through Dynamic Time Warping algorithm. Data collected through GPS devices on cruise passengers’ behavior in the port of Palermo are analyzed in order to show similarities and differences among their spatial trajectories at the destination. A cluster analysis is performed in order to identify cruise passengers’ segments based on trajectories’ similarity. Results are of interest from both a methodological perspective related with the analysis of GPS data, and for the management and planning of cruise tourism destinations.

We develop an algorithm that is fast and scalable in the detection of a nested partition extracted from a dendrogram that is obtained from hierarchical clustering of a multivariate series. Our algorithm provides a -value for each clade observed in the hierarchical tree. The -value is obtained by computing many bootstrap replicas of the dissimilarity matrix and by performing a statistical test on each difference between the dissimilarity associated with a given clade and the dissimilarity of the clade of its parent node. We prove the efficacy of our algorithm with a set of benchmarks generated by a hierarchically nested factor model. We compare results obtained by our algorithm with those of Pvclust. Pvclust is a widely-used algorithm pursuing a global approach originally developed in the context of phylogenetic studies. In our numerical experiments, we focus on the role of multiple hypothesis test correction and the robustness of the algorithms to inaccuracies and errors of datasets. We verify that our algorithm is much faster than Pvclust algorithm and has a better scalability both in the number of elements and in the number of records of the investigated multivariate set. We also apply our algorithm to two empirical datasets, one related to a biological complex system and the other related to financial time-series. We prove that the clusters detected by our methodology are meaningful with respect to some consensus partitioning of the two datasets.

We introduce a method to predict which correlation matrix coefficients are likely to change their signs in the future in the high-dimensional regime, i.e., when the number of features is larger than the number of samples per feature. The stability of correlation signs, two-by-two relationships, is found to depend on three-by-three relationships inspired by Heider social cohesion theory in this regime. We apply our method to US and Hong Kong equities historical data to illustrate how the structure of correlation matrices influences the stability of the sign of its coefficients.