Search

Quantum illumination (QI) provides entanglement-enabled target-detection enhancement, despite operating in an entanglement-breaking environment. Existing experimental studies of QI have utilized a Bayesian approach, assuming that the target is equally likely to be present or absent before detection, to demonstrate an advantage over classical target detection. However, such a premise breaks down in practical operational scenarios in which the prior probability is unknown, thereby hindering QI's applicability to real-world target-detection scenarios. In this work, we adopt the Neyman-Pearson criterion in lieu of the error probability for equally likely target absence or presence as our figure of merit for QI. We demonstrate an unconditional quantum advantage over the optimal classical-illumination protocol as benchmarked by the receiver operating characteristic, which examines detection probability versus false-alarm probability without resorting to known prior probabilities. Our work represents a critical advancement in adapting quantum-enhanced sensing to practical operational settings., Comment: 12 pages, 7 figures

The Pearson product-moment correlation coefficient (rp) and the Spearman rank correlation coefficient (rs) are widely used in psychological research. We compare rp and rs on 3 criteria: variability, bias with respect to the population value, and robustness to an outlier. Using simulations across low (N = 5) to high (N = 1,000) sample sizes we show that, for normally distributed variables, rp and rs have similar expected values but rs is more variable, especially when the correlation is strong. However, when the variables have high kurtosis, rp is more variable than rs. Next, we conducted a sampling study of a psychometric dataset featuring symmetrically distributed data with light tails, and of 2 Likert-type survey datasets, 1 with light-tailed and the other with heavy-tailed distributions. Consistent with the simulations, rp had lower variability than rs in the psychometric dataset. In the survey datasets with heavy-tailed variables in particular, rs had lower variability than rp, and often corresponded more accurately to the population Pearson correlation coefficient (Rp) than rp did. The simulations and the sampling studies showed that variability in terms of standard deviations can be reduced by about 20% by choosing rs instead of rp. In comparison, increasing the sample size by a factor of 2 results in a 41% reduction of the standard deviations of rs and rp. In conclusion, rp is suitable for light-tailed distributions, whereas rs is preferable when variables feature heavy-tailed distributions or when outliers are present, as is often the case in psychological research.

Motor imagery-based BCI systems have been promising and gaining popularity in rehabilitation and Activities of daily life(ADL). Despite this, the technology is still emerging and has not yet been outside the laboratory constraints. Channel reduction is one contributing avenue to make these systems part of ADL. Although Motor Imagery classification heavily depends on spatial factors, single channel-based classification remains an avenue to be explored thoroughly. Since Fisher's ratio and Pearson Correlation are powerful measures actively used in the domain, we propose an integrated framework (FRPC integrated framework) that integrates Fisher's Ratio to select the best channel and Pearson correlation to select optimal filter banks and extract spectral and temporal features respectively. The framework is tested for a 2-class motor imagery classification on 2 open-source datasets and 1 collected dataset and compared with state-of-art work. Apart from implementing the framework, this study also explores the most optimal channel among all the subjects and later explores classes where the single-channel framework is efficient.

Thousands of experiments are analyzed and papers are published each year involving the statistical analysis of grouped data. While this area of statistics is often perceived - somewhat naively - as saturated, several misconceptions still affect everyday practice, and new frontiers have so far remained unexplored. Researchers must be aware of the limitations affecting their analyses and what are the new possibilities in their hands. Motivated by this need, the article introduces a unifying approach to the analysis of grouped data which allows us to study the class of divisible statistics - that includes Pearson's $\chi^2$, the likelihood ratio as special cases - with a fresh perspective. The contributions collected in this manuscript span from modeling and estimation to distribution-free goodness-of-fit tests. Perhaps the most surprising result presented here is that, in a sparse regime, all tests proposed in the literature are dominated by a class of weighted linear statistics.

Reliably measuring the collinearity of bivariate data is crucial in statistics, particularly for time-series analysis or ongoing studies in which incoming observations can significantly impact current collinearity estimates. Leveraging identities from Welford's online algorithm for sample variance, we develop a rigorous theoretical framework for analyzing the maximal change to the Pearson correlation coefficient and its p-value that can be induced by additional data. Further, we show that the resulting optimization problems yield elegant closed-form solutions that can be accurately computed by linear- and constant-time algorithms. Our work not only creates new theoretical avenues for robust correlation measures, but also has broad practical implications for disciplines that span econometrics, operations research, clinical trials, climatology, differential privacy, and bioinformatics. Software implementations of our algorithms in Cython-wrapped C are made available at https://github.com/marc-harary/sensitivity for reproducibility, practical deployment, and future theoretical development., Comment: Fixed minor typos and added new citations

This paper studies the distributed conditional feature screening for massive data with ultrahigh-dimensional features. Specifically, three distributed partial correlation feature screening methods (SAPS, ACPS and JDPS methods) are firstly proposed based on Pearson partial correlation. The corresponding consistency of distributed estimation and the sure screening property of feature screening methods are established. Secondly, because using a hard threshold in feature screening will lead to a high false discovery rate (FDR), this paper develops a two-step distributed feature screening method based on knockoff technique to control the FDR. It is shown that the proposed method can control the FDR in the finite sample, and also enjoys the sure screening property under some conditions. Different from the existing screening methods, this paper not only considers the influence of a conditional variable on both the response variable and feature variables in variable screening, but also studies the FDR control issue. Finally, the effectiveness of the proposed methods is confirmed by numerical simulations and a real data analysis.

Categorical variables are of uttermost importance in biomedical research. When two of them are considered, it is often the case that one wants to test whether or not they are statistically dependent. We show weaknesses of classical methods -- such as Pearson's and the G-test -- and we propose testing strategies based on distances that lack those drawbacks. We first develop this theory for classical two-dimensional contingency tables, within the context of distance covariance, an association measure that characterises general statistical independence of two variables. We then apply the same fundamental ideas to one-dimensional tables, namely to the testing for goodness of fit to a discrete distribution, for which we resort to an analogous statistic called energy distance. We prove that our methodology has desirable theoretical properties, and we show how we can calibrate the null distribution of our test statistics without resorting to any resampling technique. We illustrate all this in simulations, as well as with some real data examples, demonstrating the adequate performance of our approach for biostatistical practice., Comment: 15 pages with 2 figures

We consider the problem of Neyman-Pearson classification which models unbalanced classification settings where error w.r.t. a distribution $\mu_1$ is to be minimized subject to low error w.r.t. a different distribution $\mu_0$. Given a fixed VC class $\mathcal{H}$ of classifiers to be minimized over, we provide a full characterization of possible distribution-free rates, i.e., minimax rates over the space of all pairs $(\mu_0, \mu_1)$. The rates involve a dichotomy between hard and easy classes $\mathcal{H}$ as characterized by a simple geometric condition, a three-points-separation condition, loosely related to VC dimension.

Recently, the first author proposed a measure to calculate Pearson correlations for node values expressed in a network, by taking into account distances or metrics defined on the network. In this technical note, we show that using an arbitrary choice of distances might result in imaginary or unbounded correlation values, which is undesired. We prove that this problem is solved by restricting to a special class of distances: negative type metrics. We also discuss two natural classes of negative type metrics on graphs, for which the network correlations are properly defined.

Not a matter of serious contention, Pearson's correlation coefficient is still the most important statistical association measure. Restricted to just two variables, this measure sometimes doesn't live up to users' needs and expectations. Specifically, a multivariable version of the correlation coefficient can greatly contribute to better assessment of the risk in a multi-asset investment portfolio. Needless to say, the correlation coefficient is derived from another concept: covariance. Even though covariance can be extended naturally by its mathematical formula, such an extension is to no use. Making matters worse, the correlation coefficient can never be extended based on its mathematical definition. In this article, we briefly explore random matrix theory to extend the notion of Pearson's correlation coefficient to an arbitrary number of variables. Then, we show that how useful this measure is at gauging noise, thereby selecting features particularly in classification., Comment: 24 pages, 40 figures

In this paper, I present three closed-form approximations of the two-sample Pearson Bayes factor. The techniques rely on some classical asymptotic results about gamma functions. These approximations permit simple closed-form calculation of the Pearson Bayes factor in cases where only the summary statistics are available (i.e., the t-score and degrees of freedom).

Organizations often rely on statistical algorithms to make socially and economically impactful decisions. We must address the fairness issues in these important automated decisions. On the other hand, economic efficiency remains instrumental in organizations' survival and success. Therefore, a proper dual focus on fairness and efficiency is essential in promoting fairness in real-world data science solutions. Among the first efforts towards this dual focus, we incorporate the equal opportunity (EO) constraint into the Neyman-Pearson (NP) classification paradigm. Under this new NP-EO framework, we (a) derive the oracle classifier, (b) propose finite-sample based classifiers that satisfy population-level fairness and efficiency constraints with high probability, and (c) demonstrate statistical and social effectiveness of our algorithms on simulated and real datasets.

Quantifying the difference between two probability density functions, $p$ and $q$, using available data, is a fundamental problem in Statistics and Machine Learning. A usual approach for addressing this problem is the likelihood-ratio estimation (LRE) between $p$ and $q$, which -- to our best knowledge -- has been investigated mainly for the offline case. This paper contributes by introducing a new framework for online non-parametric LRE (OLRE) for the setting where pairs of iid observations $(x_t \sim p, x'_t \sim q)$ are observed over time. The non-parametric nature of our approach has the advantage of being agnostic to the forms of $p$ and $q$. Moreover, we capitalize on the recent advances in Kernel Methods and functional minimization to develop an estimator that can be efficiently updated online. We provide theoretical guarantees for the performance of the OLRE method along with empirical validation in synthetic experiments.

Drasgow, Levine, and Zickar (1996) suggested a statistic based on the Neyman-Pearson lemma (e.g., Lehmann & Romano, 2005, p. 60) for detecting preknowledge on a known set of items. The statistic is a special case of the optimal appropriateness indices of Levine and Drasgow (1988) and is the most powerful statistic for detecting item preknowledge when the assumptions underlying the statistic hold for the data (e.g., Belov, 2016; Drasgow et al., 1996). This paper demonstrated using real data analysis that one assumption underlying the statistic of Drasgow et al. (1996) is often likely to be violated in practice. This paper also demonstrated, using simulated data, that the statistic is not robust to realistic violations of its underlying assumptions. Together, the results from the real data and the simulations demonstrate that the statistic of Drasgow et al. (1996) may not always be the optimum statistic in practice and occasionally has smaller power than another statistic for detecting preknowledge on a known set of items, especially when the assumptions underlying the former statistic do not hold. The findings of this paper demonstrate the importance of keeping in mind the assumptions underlying and the limitations of any statistic or method. [This paper will be published in "Applied Psychological Measurement."]

Conventionally, the total correlations within a quantum system are quantified through distance-based expressions such as the relative entropy or the square-norm. Those expressions imply that a quantum state can contain both classical and quantum correlations. In this work, we provide an alternative method to quantify the total correlations through the Pearson correlation coefficient. Using this method, we argue that a quantum state can be correlated in either a classical or a quantum way, i.e., the two cases are mutually exclusive. We also illustrate that, at least for the case of two-qubit systems, the distribution of the correlations among certain locally incompatible pairs of observables provides insight in regards to whether a system contains classical or quantum correlations. Finally, we show how correlations in quantum systems are connected to the general entropic uncertainty principle., Comment: 8 pages, 3 figures

The Pearson-Matthews correlation coefficient (usually abbreviated MCC) is considered to be one of the most useful metrics for the performance of a binary classification or hypothesis testing method (for the sake of conciseness we will use the classification terminology throughout, but the concepts and methods discussed in the paper apply verbatim to hypothesis testing as well). For multinary classification tasks (with more than two classes) the existing extension of MCC, commonly called the $\text{R}_{\text{K}}$ metric, has also been successfully used in many applications. The present paper begins with an introductory discussion on certain aspects of MCC. Then we go on to discuss the topic of multinary classification that is the main focus of this paper and which, despite its practical and theoretical importance, appears to be less developed than the topic of binary classification. Our discussion of the $\text{R}_{\text{K}}$ is followed by the introduction of two other metrics for multinary classification derived from the multivariate Pearson correlation (MPC) coefficients. We show that both $\text{R}_{\text{K}}$ and the MPC metrics suffer from the problem of not decisively indicating poor classification results when they should, and introduce three new enhanced metrics that do not suffer from this problem. We also present an additional new metric for multinary classification which can be viewed as a direct extension of MCC.

The Pearson family of ergodic diffusions with a quadratic diffusion coefficient and a linear force are characterized by explicit dynamics of their integer moments and by explicit relaxation spectral properties towards their steady state. Besides the Ornstein-Uhlenbeck process with a Gaussian steady state, the other representative examples of the Pearson family are the Square-Root or the Cox-Ingersoll-Ross process converging towards the Gamma-distribution, the Jacobi process converging towards the Beta-distribution, the reciprocal-Gamma process (corresponding to an exponential functional of the Brownian motion) that converges towards the Inverse-Gamma-distribution, the Fisher-Snedecor process, and the Student process, so that the last three steady states display heavy-tails. The goal of the present paper is to analyze the large deviations properties of these various diffusion processes in a unified framework. We first consider the Level 1 concerning time-averaged observables over a large time-window $T$ : we write the first rescaled cumulants for generic observables and we identify the specific observables whose large deviations can be explicitly computed from the dominant eigenvalue of the appropriate deformed-generator. The explicit large deviations at Level 2 concerning the time-averaged density are then used to analyze the statistical inference of model parameters from data on a very long stochastic trajectory in order to obtain the explicit rate function for the two inferred parameters of the Pearson linear force., Comment: v3=final version (63 pages) with new discussions of boundary conditions in various sections with respect to v1

COVID-19 forced parents and their children to abruptly shift to "emergency remote learning" while simultaneously juggling quarantines and new ways of working while worrying about loved ones. This report examines the past year and finds that amid a year of complex challenges, American parents found positive and unexpected benefits to living and learning during the pandemic. Pearson and Connections Academy conducted the study in partnership with Dynata to provide parents, teachers, and school systems with insight into how online learning during the pandemic -- one year later -- has evolved. 1,052 K-12 parents in the US participated in the poll which was in field from February 18, through February 26, 2021. The survey was deployed to ensure sample consistency with the first wave of the survey launched April 17-20 2020. Findings include: (1) Parents see a future with online learning; (2) Parents are concerned about learning loss and social isolation; but the majority believe most kids will "bounce back quickly" and with valuable new skills; (3) Families got closer, and men took a more active role in their children's education; (4) Recognition that women who left careers to take care of their children during the pandemic, low-income families and children of color will have a hard time regaining ground; (5) Teachers and parents hit their stride with online learning; (6) The pre-pandemic mental health issues impacting young people continue to be a problem; (7) High schoolers have pushed pause on typical teen rights-of-passage; (8) Parents and children like the flexibility and safety of online learning, but kids miss their friends and can struggle to stay focused; (9) Majority of parents are open to rethinking the structure of school; and (10) Parents expressed mixed emotions about how the introduction of the vaccine will impact what's next for school. [For the 2020 report, "The Ups & Downs: Schooling during a Pandemic. The Pearson and Connections Academy Parent Pulse Report: May 2020," see ED619658.]

The Neyman-Pearson strategy for hypothesis testing can be employed for goodness of fit if the alternative hypothesis is selected from data by exploring a rich parametrised family of models, while controlling the impact of statistical fluctuations. The New Physics Learning Machine (NPLM) methodology has been developed as a concrete implementation of this idea, to target the detection of new physical effects in the context of high energy physics collider experiments. In this paper we conduct a comparison of this approach to goodness of fit with others, in particular with classifier-based strategies that share strong similarities with NPLM. From our comparison, NPLM emerges as the more sensitive test to small departures of the data from the expected distribution and not biased towards detecting specific types of anomalies. These features make it suited for agnostic searches for new physics at collider experiments. Its deployment in other scientific and industrial scenarios should be investigated., Comment: 38 pages; improved presentation and writing throughout the paper

Reliable multi-agent trajectory prediction is crucial for the safe planning and control of autonomous systems. Compared with single-agent cases, the major challenge in simultaneously processing multiple agents lies in modeling complex social interactions caused by various driving intentions and road conditions. Previous methods typically leverage graph-based message propagation or attention mechanism to encapsulate such interactions in the format of marginal probabilistic distributions. However, it is inherently sub-optimal. In this paper, we propose IPCC-TP, a novel relevance-aware module based on Incremental Pearson Correlation Coefficient to improve multi-agent interaction modeling. IPCC-TP learns pairwise joint Gaussian Distributions through the tightly-coupled estimation of the means and covariances according to interactive incremental movements. Our module can be conveniently embedded into existing multi-agent prediction methods to extend original motion distribution decoders. Extensive experiments on nuScenes and Argoverse 2 datasets demonstrate that IPCC-TP improves the performance of baselines by a large margin., Comment: CVPR 2023 accepted

Given a random sample from a multivariate normal distribution whose covariance matrix is a Toeplitz matrix, we study the largest off-diagonal entry of the sample correlation matrix. Assuming the multivariate normal distribution has the covariance structure of an auto-regressive sequence, we establish a phase transition in the limiting distribution of the largest off-diagonal entry. We show that the limiting distributions are of Gumbel-type (with different parameters) depending on how large or small the parameter of the autoregressive sequence is. At the critical case, we obtain that the limiting distribution is the maximum of two independent random variables of Gumbel distributions. This phase transition establishes the exact threshold at which the auto-regressive covariance structure behaves differently than its counterpart with the covariance matrix equal to the identity. Assuming the covariance matrix is a general Toeplitz matrix, we obtain the limiting distribution of the largest entry under the ultra-high dimensional settings: it is a weighted sum of two independent random variables, one normal and the other following a Gumbel-type law. The counterpart of the non-Gaussian case is also discussed. As an application, we study a high-dimensional covariance testing problem.

Density Ratio Estimation (DRE) is an important machine learning technique with many downstream applications. We consider the challenge of DRE with missing not at random (MNAR) data. In this setting, we show that using standard DRE methods leads to biased results while our proposal (M-KLIEP), an adaptation of the popular DRE procedure KLIEP, restores consistency. Moreover, we provide finite sample estimation error bounds for M-KLIEP, which demonstrate minimax optimality with respect to both sample size and worst-case missingness. We then adapt an important downstream application of DRE, Neyman-Pearson (NP) classification, to this MNAR setting. Our procedure both controls Type I error and achieves high power, with high probability. Finally, we demonstrate promising empirical performance both synthetic data and real-world data with simulated missingness., Comment: 40 pages, 11 Figures. To be published in proceedings for AISTAT 2023