Search

A multivariate data analysis (MVDA) is a powerful statistical approach to simultaneously analyze datasets with multiple variables. Unlike univariate or bivariate analyses, which simultaneously focus on one or two variables, respectively, MVDA considers the interactions and relationships among multiple variables within a dataset. Several nonparametric tests can be used in the context of onesample multivariate location problems. The exact distributions of such tests cannot be analytically computed and are usually approximated using an asymptotic approximation. This article proposes the saddlepoint approximation method to approximate the tail probability for multivariate sign and signedrank tests. It is suggested as a more accurate alternative to the traditional asymptotic approximation method and an alternative to the simulation method. It requires a lot of time as it depends on all possible permutations. Real data examples were provided to illustrate the calculation of p-values, and a simulation study was conducted to compare the accuracy of the saddlepoint approximation method with the simulation method (permutation-based, so time-consuming) and an asymptotic normal approximation method. The study results show that the saddlepoint approximation provides highly accurate approximations to the p-values of the considered statistics, and it often outperforms the normal approximation. Additionally, the results show that the proposed method’s computation time is much less than that of the time-consuming simulation method. [ABSTRACT FROM AUTHOR]

AbstractBivariate analysis is essential in several applied fields, such as medicine, engineering, biology, and econometrics. Many parametric and non-parametric procedures can be used for bivariate analysis. Non-parametric procedures require fewer conditions than their parametric counterparts. Therefore, this article proposes an accurate approximation of the p-values and the probability mass functions of some non-parametric bivariate tests, including bivariate sign tests and medians. The proposed approximation, saddlepoint approximation, is compared to the traditional approximation method, asymptotic normal approximation method, and Edgeworth expansion through three real data examples and a simulation study. The numerical results show that the proposed approximation method is more accurate than the asymptotic method and Edgeworth expansion. Furthermore, it is computationally less demanding than the simulation method, which is permutation-based and time-consuming. [ABSTRACT FROM AUTHOR]

We develop a new permutation test for inference on a subvector of coefficients in linear models. The test is exact when the regressors and the error terms are independent. Then we show that the test is asymptotically of correct level, consistent, and has power against local alternatives when the independence condition is relaxed, under two main conditions. The first is a slight reinforcement of the usual absence of correlation between the regressors and the error term. The second is that the number of strata, defined by values of the regressors not involved in the subvector test, is small compared to the sample size. The latter implies that the vector of nuisance regressors is discrete. Simulations and empirical illustrations suggest that the test has good power in practice if, indeed, the number of strata is small compared to the sample size. [ABSTRACT FROM AUTHOR]

The meaning of randomization tests has become obscure in statistics education and practice over the last century. This article makes a fresh attempt at rectifying this core concept of statistics. A new term—"quasi-randomization test"—is introduced to define significance tests based on theoretical models and distinguish these tests from the "randomization tests" based on the physical act of randomization. The practical importance of this distinction is illustrated through a real stepped-wedge cluster-randomized trial. Building on the recent literature on randomization inference, a general framework of conditional randomization tests is developed and some practical methods to construct conditioning events are given. The proposed terminology and framework are then applied to understand several widely used (quasi-)randomization tests, including Fisher's exact test, permutation tests for treatment effect, quasi-randomization tests for independence and conditional independence, adaptive randomization, and conformal prediction. for this article are available online. [ABSTRACT FROM AUTHOR]

Inference on inequality indices remains challenging, even in large samples. Heavy right tails in income and wealth distributions hinder the quality and threaten the validity of asymptotic approximations to finite sample distributions. Attempts to improve on asymptotic approximations by bootstrap techniques or permutation tests are only partial successes. We evaluate a different approach to robust inference, relying on Student t statistics obtained from split samples. This relatively simple 't-based' approach requires no consistent variance estimators, no random sampling of populations, and only mild distributional assumptions. We compare its performance with that of refined bootstrap and permutation techniques. We find that the more complex bootstrap methods still have the edge in one-sample tests, where the t-approach suffers from a negative skew. In two-sample comparisons though, the t-approach offers advantages: it is undersized while bootstrap tests and permutation tests are often oversized. In certain circumstances it is less powerful than permutation tests and bootstrap tests, but for large samples, this difference dissipates. It is also more generally applicable than permutation tests and easily generates confidence intervals. These differences are illustrated with an empirical application using two different sources of household data from the Russian Federation. [ABSTRACT FROM AUTHOR]

Objectives: We reconstruct the networks of officers co-involved in force incidents to test whether interactions with weapon-prone peers impact firearm use. Methods: We draw from a statewide dataset of force incidents across law enforcement agencies in New Jersey, and employ conditional likelihood models to estimate whether exposure to peers with histories of firearm use is associated with an officer's own likelihood of firearm use net of other contextual confounders. Results: We find preliminary evidence that officer firearm behaviors, including drawing, pointing, and discharging a firearm, is influenced by an officer's peers. Greater exposure to colleagues with histories of firearm use is associated with a lower risk of using a firearm. We also find that officer features, including experience and race/ethnicity, are associated with the risk of firearm use. Conclusions: Our study suggests officers' peers structure the risk of firearm use. Our data allow us to look at time order and rule out situational confounders pertaining to firearm use; however, do not allow us to infer causality. We discuss the study's implications for understanding firearm behaviors and the role of network science in moving policing research forward. [ABSTRACT FROM AUTHOR]

In the evolving landscape of psycholinguistic research, this study addresses the inherent complexities of data through advanced analytical methodologies, including permutation tests, bootstrap confidence intervals, and fractile or quantile regression. The methodology and philosophy of our approach deeply resonate with fractal and fractional concepts. Responding to the skewed distributions of data, which are observed in metrics such as reading times, time-to-response, and time-to-submit, our analysis highlights the nuanced interplay between time-to-response and variables like lists, conditions, and plausibility. A particular focus is placed on the implausible sentence response times, showcasing the precision of our chosen methods. The study underscores the profound influence of individual variability, advocating for meticulous analytical rigor in handling intricate and complex datasets. Drawing inspiration from fractal and fractional mathematics, our findings emphasize the broader potential of sophisticated mathematical tools in contemporary research, setting a benchmark for future investigations in psycholinguistics and related disciplines. [ABSTRACT FROM AUTHOR]

Abstract Survival analysis is most typically characterized by left truncation; left-truncated data are composed of two recorded times, the time of occurrence of an event before the event of interest, and the time of the interested event. In this study, p-values of the permutation distribution of the linear rank tests are computed using saddlepoint approximation method for left-truncated data using a randomized block design. Simulation studies demonstrate the saddlepoint approximation method’s accuracy in approximating p-value of the linear rank tests. Due to the efficiency of the proposed method over the normal approximation, 95% confidence intervals for the treatment impact are performed. [ABSTRACT FROM AUTHOR]

Permutation tests are the standard technique for significance testing in Analysis of Variance Simultaneous Component Analysis. However, there is a vast number of alternative approaches for permutation testing, and the number of choices grows in relation to the complexity of the study design. In this paper, we focus on longitudinal intervention studies with multivariate outcomes, a relevant experimental design in clinical studies where the outcome is an omics profile (such as in genomics, metabolomics, and the like). We propose a new technique to derive power curves tailored to the size and (un)balanced nature of the data set in the study. This technique is useful to identify misleading permutation tests, with lack of power or overly optimistic outcomes. We found that choosing the best permutation approach is far from intuitive and that there is a significant risk of deriving incorrect conclusions in real‐life analyses. Our approach avoids this risk and can be extended to other complex designs of interest. The code is available for free use. Permutation tests are the standard technique for significance testing in ASCA. However, there is a vast number of alternative approaches. In this paper, we focus on clinical studies where the outcome is an omics profile (such as in metabolomics and the like). We propose a new technique to derive power curves tailored to the size and (un)balanced nature of the data set. This technique is useful to identify misleading permutation tests, with lack of power or overly optimistic outcomes. [ABSTRACT FROM AUTHOR]

Left truncation is a feature of many clinical trials and survival analysis in which lifetimes are observed conditionally on being greater than random truncation times. The subjects of such studies are exposed to an event prior to the event of interest and both times are recorded for each subject. In other words, left-truncated data arise when we only observe these subjects whose failure event times are limited below by truncation event times. For such type of data, the most commonly used class of two-sample tests is the weighted log-rank class. The aim of this article is to use saddlepoint approximation techniques to calculate p-values from the null permutation distributions of tests from the weighted log-rank class. The saddlepoint p-values are almost closer to the exact mid-p-values. The weighted log-rank tests are inverted to determine nominal 95% confidence intervals for the treatment effect in the presence of left-truncated data. Such analytical inversions lead to permutation confidence intervals which are easily computed and virtually identical with the exact intervals that would usually require huge amounts of simulations. [ABSTRACT FROM AUTHOR]

In clinical trials, Clustered data with censored failure times often arise from a litter-matched tumorigenesis experiment. Based on such data, the mutual and widely used class of two-sample tests is the weighted log-rank tests. A double saddle-point approximation is used to calculate the p-values of the null permutation distribution of these tests. This approximation is superior to the asymptotic normal approximation. This precision allows us to determine exact confidence intervals for the treatment impact. The findings reveal the efficiency of saddle-point approximation using two real clustered data sets. Extensive simulation studies are carried out to assess the performance of the saddle-point approximation. [ABSTRACT FROM AUTHOR]

Abstract Background The HEALing (Helping to End Addiction Long-termSM) Communities Study (HCS) is a multi-site parallel group cluster randomized wait-list comparison trial designed to evaluate the effect of the Communities That Heal (CTH) intervention compared to usual care on opioid overdose deaths. Covariate-constrained randomization (CCR) was applied to balance the community-level baseline covariates in the HCS. The purpose of this paper is to evaluate the performance of model-based tests and permutation tests in the HCS setting. We conducted a simulation study to evaluate type I error rates and power for model-based and permutation tests for the multi-site HCS as well as for a subgroup analysis of a single state (Massachusetts). We also investigated whether the maximum degree of imbalance in the CCR design has an impact on the performance of the tests. Methods The primary outcome, the number of opioid overdose deaths, is count data assessed at the community level that will be analyzed using a negative binomial regression model. We conducted a simulation study to evaluate the type I error rates and power for 3 tests: (1) Wald-type t-test with small-sample corrected empirical standard error estimates, (2) Wald-type z-test with model-based standard error estimates, and (3) permutation test with test statistics calculated by the difference in average residuals for the two groups. Results Our simulation results demonstrated that Wald-type t-tests with small-sample corrected empirical standard error estimates from the negative binomial regression model maintained proper type I error. Wald-type z-tests with model-based standard error estimates were anti-conservative. Permutation tests preserved type I error rates if the constrained space was not too small. For all tests, the power was high to detect the hypothesized 40% reduction in opioid overdose deaths for the intervention vs. comparison group both for the overall HCS and the subgroup analysis of Massachusetts (MA). Conclusions Based on the results of our simulation study, the Wald-type t-test with small-sample corrected empirical standard error estimates from a negative binomial regression model is a valid and appropriate approach for analyzing cluster-level count data from the HEALing Communities Study. Trial registration ClinicalTrials.gov http://www.clinicaltrials.gov ; Identifier: NCT04111939

Clustered data frequently occur in biomedical research fields and clinical trials. The log-rank tests are widely used for two-independent samples of clustered data tests. The randomized block design and truncated binomial design are used for forcing balance in clinical trials and reducing selection bias. In this paper, survival clustered data are randomized by generalized randomized block, and subsequently clustered data in each block are randomized by truncated binomial design. Consequently, the p-values of the null permutation distribution of log-rank tests for clustered data are approximated via the double saddlepoint approximation method. Comprehensive numerical studies are carried out to assess the accuracy of the saddlepoint approximation. This approximation has a great accuracy over the asymptotic normal approximation. [ABSTRACT FROM AUTHOR]

Clustered survival data frequently occurs in biomedical research fields and clinical trials. The log-rank tests are used for two independent samples of clustered data tests. We use the block Efron's biased-coin randomization (design) to assign patients to treatment groups in a clinical trial by forcing a sequential experiment to be balanced. In this article, the p -values of the null permutation distribution of log-rank tests for clustered data are approximated via the double saddlepoint approximation method. Comprehensive numerical studies are carried out to assess the accuracy of the saddlepoint approximation. This approximation demonstrates great accuracy over the asymptotic normal approximation. [ABSTRACT FROM AUTHOR]

We propose using a permutation test to detect discontinuities in an underlying economic model at a known cutoff point. Relative to the existing literature, we show that this test is well suited for event studies based on time‐series data. The test statistic measures the distance between the empirical distribution functions of observed data in two local subsamples on the two sides of the cutoff. Critical values are computed via a standard permutation algorithm. Under a high‐level condition that the observed data can be coupled by a collection of conditionally independent variables, we establish the asymptotic validity of the permutation test, allowing the sizes of the local subsamples to be either be fixed or grow to infinity. In the latter case, we also establish that the permutation test is consistent. We demonstrate that our high‐level condition can be verified in a broad range of problems in the infill asymptotic time‐series setting, which justifies using the permutation test to detect jumps in economic variables such as volatility, trading activity, and liquidity. These potential applications are illustrated in an empirical case study for selected FOMC announcements during the ongoing COVID‐19 pandemic. [ABSTRACT FROM AUTHOR]

The non-independence of social network data is a cause for concern among behavioural ecologists conducting social network analysis. This has led to the adoption of several permutation-based methods for testing common hypotheses. One of the most common types of analysis is nodal regression, where the relationships between node-level network metrics and nodal covariates are analysed using a permutation technique known as node-label permutations. We show that, contrary to accepted wisdom, node-label permutations do not automatically account for the non-independences assumed to exist in network data, because regression-based permutation tests still assume exchangeability of residuals. The same assumption also applies to the quadratic assignment procedure (QAP), a permutation-based method often used for conducting dyadic regression. We highlight that node-label permutations produce the same p-values as equivalent parametric regression models, but that in the presence of non-independence, parametric regression models can also produce accurate effect size estimates. We also note that QAP only controls for a specific type of non-independence between edges that are connected to the same nodes, and that appropriate parametric regression models are also able to account for this type of non-independence. Based on this, we suggest that standard parametric models could be used in the place of permutation-based methods. Moving away from permutation-based methods could have several benefits, including reducing over-reliance on p-values, generating more reliable effect size estimates, and facilitating the adoption of causal inference methods and alternative types of statistical analysis. [ABSTRACT FROM AUTHOR]

A connection between the general linear model (GLM) with frequentist statistical testing and machine learning (MLE) inference is derived and illustrated. Initially, the estimation of GLM parameters is expressed as a Linear Regression Model (LRM) of an indicator matrix; that is, in terms of the inverse problem of regressing the observations. Both approaches, i.e. GLM and LRM, apply to different domains, the observation and the label domains, and are linked by a normalization value in the least-squares solution. Subsequently, we derive a more refined predictive statistical test: the linear Support Vector Machine (SVM), that maximizes the class margin of separation within a permutation analysis. This MLE-based inference employs a residual score and associated upper bound to compute a better estimation of the actual (real) error. Experimental results demonstrate how parameter estimations derived from each model result in different classification performance in the equivalent inverse problem. Moreover, using real data, the MLE-based inference including model-free estimators demonstrates an efficient trade-off between type I errors and statistical power. [ABSTRACT FROM AUTHOR]

In the evolving landscape of psycholinguistic research, this study addresses the inherent complexities of data through advanced analytical methodologies, including permutation tests, bootstrap confidence intervals, and fractile or quantile regression. The methodology and philosophy of our approach deeply resonate with fractal and fractional concepts. Responding to the skewed distributions of data, which are observed in metrics such as reading times, time-to-response, and time-to-submit, our analysis highlights the nuanced interplay between time-to-response and variables like lists, conditions, and plausibility. A particular focus is placed on the implausible sentence response times, showcasing the precision of our chosen methods. The study underscores the profound influence of individual variability, advocating for meticulous analytical rigor in handling intricate and complex datasets. Drawing inspiration from fractal and fractional mathematics, our findings emphasize the broader potential of sophisticated mathematical tools in contemporary research, setting a benchmark for future investigations in psycholinguistics and related disciplines.

There remains an open question about the usefulness and the interpretation of machine learning (ML) approaches for discrimination of spatial patterns of brain images between samples or activation states. In the last few decades, these approaches have limited their operation to feature extraction and linear classification tasks for between-group inference. In this context, statistical inference is assessed by randomly permuting image labels or by the use of random effect models that consider between-subject variability. These multivariate ML-based statistical pipelines, whilst potentially more effective for detecting activations than hypotheses-driven methods, have lost their mathematical elegance, ease of interpretation, and spatial localization of the ubiquitous General linear Model (GLM). Recently, the estimation of the conventional GLM parameters has been demonstrated to be connected to an univariate classification task when the design matrix in the GLM is expressed as a binary indicator matrix. In this paper we explore the complete connection between the univariate GLM and ML-based regressions. To this purpose we derive a refined statistical test with the GLM based on the parameters obtained by a linear Support Vector Regression (SVR) in the inverse problem (SVR-iGLM). Subsequently, random field theory (RFT) is employed for assessing statistical significance following a conventional GLM benchmark. Experimental results demonstrate how parameter estimations derived from each model (mainly GLM and SVR) result in different experimental design estimates that are significantly related to the predefined functional task. Moreover, using real data from a multisite initiative the proposed ML-based inference demonstrates statistical power and the control of false positives, outperforming the regular GLM. [ABSTRACT FROM AUTHOR]

Background: The HEALing (Helping to End Addiction Long-termSM) Communities Study (HCS) is a multi-site parallel group cluster randomized wait-list comparison trial designed to evaluate the effect of the Communities That Heal (CTH) intervention compared to usual care on opioid overdose deaths. Covariate-constrained randomization (CCR) was applied to balance the community-level baseline covariates in the HCS. The purpose of this paper is to evaluate the performance of model-based tests and permutation tests in the HCS setting. We conducted a simulation study to evaluate type I error rates and power for model-based and permutation tests for the multi-site HCS as well as for a subgroup analysis of a single state (Massachusetts). We also investigated whether the maximum degree of imbalance in the CCR design has an impact on the performance of the tests.Methods: The primary outcome, the number of opioid overdose deaths, is count data assessed at the community level that will be analyzed using a negative binomial regression model. We conducted a simulation study to evaluate the type I error rates and power for 3 tests: (1) Wald-type t-test with small-sample corrected empirical standard error estimates, (2) Wald-type z-test with model-based standard error estimates, and (3) permutation test with test statistics calculated by the difference in average residuals for the two groups.Results: Our simulation results demonstrated that Wald-type t-tests with small-sample corrected empirical standard error estimates from the negative binomial regression model maintained proper type I error. Wald-type z-tests with model-based standard error estimates were anti-conservative. Permutation tests preserved type I error rates if the constrained space was not too small. For all tests, the power was high to detect the hypothesized 40% reduction in opioid overdose deaths for the intervention vs. comparison group both for the overall HCS and the subgroup analysis of Massachusetts (MA).Conclusions: Based on the results of our simulation study, the Wald-type t-test with small-sample corrected empirical standard error estimates from a negative binomial regression model is a valid and appropriate approach for analyzing cluster-level count data from the HEALing Communities Study.Trial Registration: ClinicalTrials.gov http://www.Clinicaltrials: gov ; Identifier: NCT04111939. [ABSTRACT FROM AUTHOR]

Left-truncated data are constructed from the time of an initial event and the time of the event of interest. That is to say, each individual represented by such data is exposed to an event prior to the event of interest, and both times are recorded. Weighted log-rank testing is commonly used for such data. In this paper, a saddlepoint approximation method is provided for computing p-values of the permutation distribution of tests from the weighted log-rank testing in the presence of left-truncated data and Wei's urn design. A simulation study is used to assess the efficiency of the saddlepoint approximation. The accuracy of the saddlepoint approximation in comparison to the normal approximation enables us to compute accurate confidence intervals for the treatment effect. [ABSTRACT FROM AUTHOR]

In this study, we focus on mixed data which are either observations of univariate random variables which can be quantitative or qualitative, or observations of multivariate random variables such that each variable can include both quantitative and qualitative components. We first propose a novel method, called CMIh, to estimate conditional mutual information taking advantages of the previously proposed approaches for qualitative and quantitative data. We then introduce a new local permutation test, called LocAT for local adaptive test, which is well adapted to mixed data. Our experiments illustrate the good behaviour of CMIh and LocAT, and show their respective abilities to accurately estimate conditional mutual information and to detect conditional (in)dependence for mixed data. [ABSTRACT FROM AUTHOR]

A new nonparametric test of equality of two densities is investigated. The test statistic is an average of log-Bayes factors, each of which is constructed from a kernel density estimate. Prior densities for the bandwidths of the kernel estimates are required, and it is shown how to choose priors so that the log-Bayes factors can be calculated exactly. Critical values of the test statistic are determined by a permutation distribution, conditional on the data. An attractive property of the methodology is that a critical value of 0 leads to a test for which both type I and II error probabilities tend to 0 as sample sizes tend to ∞. Existing results on Kullback–Leibler loss of kernel estimates are crucial to obtaining these asymptotic results, and also imply that the proposed test works best with heavy-tailed kernels. Finite sample characteristics of the test are studied via simulation, and extensions to multivariate data are straightforward, as illustrated by an application to bivariate connectionist data. [ABSTRACT FROM AUTHOR]

Diversity analysis is a de facto standard procedure for most existing microbiome studies. Nevertheless, diversity metrics can be insensitive to changes in community composition (identities). For example, if species A (e.g., a beneficial microbe) is replaced by equal number of species B (e.g., an opportunistic pathogen), the diversity metric may not change, but the community composition has changed. The shared species analysis (SSA) is a computational technique that can discern changes of community composition by detecting the increase/decrease of shared species between two sets of microbiome samples, and it should be more sensitive than standard diversity analysis in discerning changes in microbiome structures. Here, we investigated the effects of ethnicity and lifestyles in China on the structure of Chinese gut microbiomes by reanalyzing the datasets of a large Chinese cohort with 300+ individuals covering 7 biggest Chinese ethnic groups (>95% Chinese population). We found: (i) Regarding lifestyles, SSA revealed significant differences between 100% of pair-wise comparisons in community compositions across all but phylum taxon levels (phylum level = 29%), but diversity analysis only revealed 14--29% pair-wise differences in community diversity across all four taxon levels. (ii) Regarding ethnicities, SSA revealed 100% pair-wise differences in community compositions across all but phylum (phylum level = 48--62%) levels, but diversity analysis only revealed 5--57% differences in community diversity across all four taxon levels. (iii) Ethnicity seems to have more prevalent effects on community structures than lifestyle does (iv) Community structures of the gut microbiomes are more stable at the phylum level than at the other three levels. (v) SSA is more powerful than diversity analysis in detecting the changes of community structures; furthermore, SSA can produce lists of unique and shared OTUs. (vi) Finally, we performed stochasticity analysis to mechanistically interpret the observed differences revealed by the SSA and diversity analyses. [ABSTRACT FROM AUTHOR]

Many metaheuristic approaches are inherently stochastic. In order to compare such methods, statistical tests are needed. However, choosing an appropriate test is not trivial, given that each test has some assumptions about the distribution of the underlying data that must be true before it can be used. Permutation tests (P-Tests) are statistical tests with minimal number of assumptions. These tests are simple, intuitive and nonparametric. In this paper, we argue researchers in the field of metaheuristics to adopt P-Tests to compare their algorithms. We define two statistic tests and then present an algorithm that uses them to compute the p-value. The proposed process is used to compare 5 metaheuristic algorithms on 10 benchmark functions. The resulting p-values are compared with the p-values of two widely used statistical tests. The results show that the proposed P-test is generally consistent with the classical tests, but more conservative in few cases. [ABSTRACT FROM AUTHOR]

Across the social sciences, scholars regularly pool effects over substantial periods of time, a practice that produces faulty inferences if the underlying data generating process is dynamic. To help researchers better perform principled analyses of time-varying processes, we develop a two-stage procedure based upon techniques for permutation testing and statistical process monitoring. Given time series cross-sectional data, we break the role of time through permutation inference and produce a null distribution that reflects a time-invariant data generating process. The null distribution then serves as a stable reference point, enabling the detection of effect changepoints. In Monte Carlo simulations, our randomization technique outperforms alternatives for changepoint analysis. A particular benefit of our method is that, by establishing the bounds for time-invariant effects before interacting with actual estimates, it is able to differentiate stochastic fluctuations from genuine changes. We demonstrate the method's utility by applying it to a popular study on the relationship between alliances and the initiation of militarized interstate disputes. The example illustrates how the technique can help researchers make inferences about where changes occur in dynamic relationships and ask important questions about such changes. [ABSTRACT FROM AUTHOR]

In this article, we present single-case causal mediation analysis as the application of causal mediation analysis to data collected within a single-case experiment. This method combines the focus on the individual with the focus on mechanisms of change, rendering it a promising approach for both mediation and single-case researchers. For this purpose, we propose a new method based on time-discrete state-space modeling to estimate the direct and indirect treatment effects. We demonstrate how to estimate the model for a single-case experiment on stress and craving in a routine alcohol consumer before and after an imposed period of abstinence. Furthermore, we present a simulation study that examines the estimation and testing of the standardized indirect effect. All parameters used to generate the data were recovered with acceptable precision. We use maximum likelihood and permutation procedures to calculate p -values and standard errors of the parameters estimates. The new method is promising for testing mediated effects in single-case experimental designs. We further discuss limitations of the new method with respect to causal inference, as well as more technical concerns, such as the choice of the time lags between the measurements. [ABSTRACT FROM AUTHOR]

In response to the importance of individual-level effects, the purpose of this paper is to describe the new randomization permutation (RP) test for a mediation mechanism for a single subject. We extend seminal work on permutation tests for individual-level data by proposing a test for mediation for one person. The method requires random assignment to the levels of the treatment variable at each measurement occasion, and repeated measures of the mediator and outcome from one subject. If several assumptions are met, the process by which a treatment changes an outcome can be statistically evaluated for a single subject, using the permutation mediation test method and the permutation confidence interval method for residuals. A simulation study evaluated the statistical properties of the new method suggesting that at least eight repeated measures are needed to control Type I error rates and larger sample sizes are needed for power approaching.8 even for large effects. The RP mediation test is a promising method for elucidating intraindividual processes of change that may inform personalized medicine and tailoring of process-based treatments for one subject. [ABSTRACT FROM AUTHOR]

Throughout the last decade, random forests have established themselves as among the most accurate and popular supervised learning methods. While their black-box nature has made their mathematical analysis difficult, recent work has established important statistical properties like consistency and asymptotic normality by considering subsampling in lieu of bootstrapping. Though such results open the door to traditional inference procedures, all formal methods suggested thus far place severe restrictions on the testing framework and their computational overhead often precludes their practical scientific use. Here we propose a hypothesis test to formally assess feature significance, which uses permutation tests to circumvent computationally infeasible estimates of nuisance parameters. This test is intended to be analogous to the F-test for linear regression. We establish asymptotic validity of the test via exchangeability arguments and show that the test maintains high power with orders of magnitude fewer computations. Importantly, the procedure scales easily to big data settings where large training and testing sets may be employed, conducting statistically valid inference without the need to construct additional models. Simulations and applications to ecological data, where random forests have recently shown promise, are provided. [ABSTRACT FROM AUTHOR]

Permutation tests are widely used to test null hypotheses with animal social network data, but suffer from high rates of type I and II error when the permutations do not properly simulate the intended null hypothesis.Two common types of permutations each have limitations. Pre‐network (or datastream) permutations can be used to control 'nuisance effects' like spatial, temporal or sampling biases, but only when the null hypothesis assumes random social structure. Node (or node‐label) permutation tests can test null hypotheses that include nonrandom social structure, but only when nuisance effects do not shape the observed network.We demonstrate one possible solution addressing these limitations: using pre‐network permutations to adjust the values for each node or edge before conducting a node permutation test. We conduct a range of simulations to estimate error rates caused by confounding effects of social or non‐social structure in the raw data.Regressions on simulated datasets suggest that this 'double permutation' approach is less likely to produce elevated error rates relative to using only node permutations, pre‐network permutations or node permutations with simple covariates, which all exhibit elevated type I errors under at least one set of simulated conditions. For example, in scenarios where type I error rates from pre‐network permutation tests exceed 30%, the error rates from double permutation remain at 5%.The double permutation procedure provides one potential solution to issues arising from elevated type I and type II error rates when testing null hypotheses with social network data. We also discuss alternative approaches that can provide robust inference, including fitting mixed effects models, restricted node permutations, testing multiple null hypotheses and splitting large datasets to generate replicated networks. Finally, we highlight ways that uncertainty can be explicitly considered and carried through the analysis. [ABSTRACT FROM AUTHOR]

Permutation tests are widely used in practice. However, these tests either need restrictive assumptions for validity, or are not applicable to high-dimensional data. This study considers permutation tests for high-dimensional mean comparisons. Here, in order to get around these restrictions, the test statistics are calculated based on pseudo samples generated using a "binning" procedure. The corresponding permutation tests are proved to be asymptotically consistent. We also consider a related problem for signal identification and establish the asymptotic properties of the tests. Simulation studies demonstrate the favorable performance of our methods compared with that of existing tests. Finally, the proposed method is applied to a genome-wide association study for seven complex human diseases to identify possible single nucleotide polymorphisms associated with the diseases. [ABSTRACT FROM AUTHOR]

Recent replication crisis has led to a number of ad hoc suggestions to decrease the chance of making false positive findings. Among them, Johnson (Proceedings of the National Academy of Sciences, 110, 19313–19317, 2013) and Benjamin et al. (Nature Human Behaviour, 2, 6–10 2018) recommend using the significance level of α = 0.005 (0.5%) as opposed to the conventional 0.05 (5%) level. Even though their suggestion is easy to implement, it is unclear whether or not the commonly used statistical tests are robust and/or powerful at such a small significance level. Therefore, the main aim of our study is to investigate the robustness and power curve behaviors of independent (unpaired) two-sample tests for metric and ordinal data at nominal significance levels of α = 0.005 and α = 0.05. Through an extensive simulation study, it is found that the permutation versions of the Welch t-test and the Brunner-Munzel test are particularly robust and powerful while the commonly used two-sample tests which utilize t-distribution tend to be either liberal or conservative, and have peculiar power curve behaviors under skewed distributions with variance heterogeneity. [ABSTRACT FROM AUTHOR]

Population means and standard deviations are the most common estimands to quantify effects in factorial layouts. In fact, most statistical procedures in such designs are built toward inferring means or contrasts thereof. For more robust analyses, we consider the population median, the interquartile range (IQR) and more general quantile combinations as estimands in which we formulate null hypotheses and calculate compatible confidence regions. Based upon simultaneous multivariate central limit theorems and corresponding resampling results, we derive asymptotically correct procedures in general, potentially heteroscedastic, factorial designs with univariate endpoints. Special cases cover robust tests for the population median or the IQR in arbitrary crossed one-, two- and higher-way layouts with potentially heteroscedastic error distributions. In extensive simulations, we analyze their small sample properties and also conduct an illustrating data analysis comparing children's height and weight from different countries. [ABSTRACT FROM AUTHOR]

This paper considers the problem of making inferences about the effects of a program on multiple outcomes when the assignment of treatment status is imperfectly randomized. By imperfect randomization we mean that treatment status is reassigned after an initial randomization on the basis of characteristics that may be observed or unobserved by the analyst. We develop a partial identification approach to this problem that makes use of information limiting the extent to which randomization is imperfect to show that it is still possible to make nontrivial inferences about the effects of the program in such settings. We consider a family of null hypotheses in which each null hypothesis specifies that the program has no effect on one of several outcomes of interest. Under weak assumptions, we construct a procedure for testing this family of null hypotheses in a way that controls the familywise error rate - the probability of even one false rejection - in finite samples. We develop our methodology in the context of a reanalysis of the HighScope Perry Preschool program. We find statistically significant effects of the program on a number of different outcomes of interest, including outcomes related to criminal activity for males and females, even after accounting for the imperfectness of the randomization and the multiplicity of null hypotheses.

The notion of testing for equivalence of two treatments is widely used in clinical trials, pharmaceutical experiments, bioequivalence and quality control. It is traditionally operated within the intersection–union principle (IU). According to this principle the null hypothesis is stated as the set of effects the differences δ of which lie outside a suitable equivalence interval and the alternative as the set of δ that lie inside it. In the literature related solutions are essentially based on likelihood techniques, which in turn are rather difficult to deal with. A recently published paper goes beyond most of likelihood limitations by using the IU approach within the permutation theory. One more paper, based on Roy's union–intersection principle (UI) within the permutation theory, goes beyond some limitations of traditional two-sided tests. Such UI approach, effectively a mirror image of IU, assumes a null hypothesis where δ lies inside the equivalence interval and an alternative where it lies outside. Since testing for equivalence can rationally be analyzed by both principles but, as the two differ in terms of the mirror-like roles assigned to the hypotheses under study, they are not strictly comparable. The present paper's main goal is to look into these problems and provide a sort of comparative analysis of both by highlighting the related requirements, properties, limitations, difficulties, and pitfalls so as to get practitioners properly acquainted with their correct use in practical contexts. [ABSTRACT FROM AUTHOR]