Search

Data for statistical analysis is often available from different samples, with each sample containing measurements on only some of the variables of interest. Statistical matching attempts to generate a fused database containing matched measurements on all the target variables. In this article, we consider the use of statistical matching when the samples are drawn by informative sampling designs and are subject to not missing at random non-response. The problem with ignoring the sampling process and non-response is that the distribution of the data observed for the responding units can be very different from the distribution holding for the population data, which may distort the inference process and result in a matched database that misrepresents the joint distribution in the population. Our proposed methodology employs the empirical likelihood approach and is shown to perform well in a simulation experiment and when applied to real sample data.

A measure of interrater absolute agreement for ordinal scales is proposed capitalizing on the dispersion index for ordinal variables proposed by Giuseppe Leti. The procedure allows to overcome the limits affecting traditional measures of interrater agreement in different fields of application. An unbiased estimator of the proposed measure is introduced and its sampling properties are investigated. In order to construct confidence intervals for interrater absolute agreement both asymptotic results and bootstrapping methods are used and their performance is evaluated. Simulated data are employed to demonstrate the accuracy and practical utility of the new procedure for assessing agreement. Finally, an application to a real case is provided.

Nowadays there is increasing availability of good quality official statistics data. The construction of multivariate statistical models possibly leading to the identification of causal relationships is of interest. In this context Bayesian networks play an important role. A crucial step consists in learning the structure of a Bayesian net- work. One of the most widely used procedures is the PC algorithm consisting in carrying out several independence tests on the available data set and in building a Bayesian network according to the tests results. The PC algorithm is based on the irremissible assumption that data are independent and identically distributed. Unfortunately, official statistics data are generally collected through complex sampling designs, then the aforementioned assumption is not met. In such a context the PC algorithm fails in learning the structure. To avoid this, the sample selection must be taken into account in the structural learning process. In this paper, a modified version of the PC algorithm is proposed for inferring causal structure from complex survey data. It is based on resampling techniques for finite populations. A simulation experiment showing the robustness with respect to departures from the assumptions and the good performance of the proposed algorithm is carried out.

Statistical matching deals with the problem of how to combine information collected in different samples taken from the same target population, but on partly different survey variables. The purpose of this paper is to analyze the statistical matching problem under informative sampling designs, when applying the sample likelihood approach. First, a conditional independence assumption is made, which allows to define an identifiable population model under which the conditions guaranteeing the identifiability and estimability of the sample likelihood are investigated. Next, the uncertainty in statistical matching under informative sampling designs is discussed, with particular attention to the three-variate normal case. A simulation experiment illustrating the theoretical results is performed.

The goal of statistical matching, at a micro level, is the construction of a synthetic data source where all the variables of interest are available. In this paper we propose the use of Bayesian Networks to deal with the statistical matching for multivariate categorical variables in the micro approach. Its performance is evaluated by an application to a real data set.

The goal of statistical matching, at a macro level, is the estimation of the joint distribution of variables separately observed in independent samples. The lack of joint information on the variables of interest leads to uncertainty about the data generating model. In this paper we propose the use of graphical models to deal with the statistical matching uncertainty for multivariate categorical variables. The use of Bayesian networks in the statistical matching context allows both to introduce extra sample information on the dependence structure between the variables of interest and to use such an information to factorize the joint probability distribution according to the graph decomposition of a multivariate dependence in lower dimension components. This representation of the joint probability distribution, taking advantage of local relationships, allows to simplify both parameters estimation and statistical matching quality evaluation in a multivariate context. A simulation experiment is performed in order to evaluate the performance of the proposed methodology with and without auxiliary information, as well as to compare it with the saturated multinomial model, in terms of uncertainty reduction. Finally, an application to a real case is provided. Results show a considerable improvement in the quality of statistical matching when the dependence structure is taken into account.

The goal of statistical matching is the estimation of a joint distribution having observed only samples from its marginals. The lack of joint observations on the variables of interest is the reason of uncertainty about the joint population distribution function. In the present article, the notion of matching error is introduced, and upper-bounded via an appropriate measure of uncertainty. Then, an estimate of the distribution function for the variables not jointly observed is constructed on the basis of a modification of the conditional independence assumption in the presence of logical constraints. The corresponding measure of uncertainty is estimated via sample data. Finally, a simulation study is performed, and an application to a real case is provided. Supplementary materials for this article are available online.

Statistical matching consists in estimating the joint characteristics of two variables observed in two distinct and independent sample surveys, respectively. In a parametric setup, ranges of estimates for non identifiable parameters are the only estimable items, unless restrictive assumptions on the probabilistic relationship between the non jointly observed variables are imposed. These ranges correspond to the uncertainty due to the absence of joint observations on the pair of variables of interest. The aim of this paper is to analyze the uncertainty in statistical matching in a non parametric setting. A measure of uncertainty is introduced, and its properties studied: this measure studies the “intrinsic” association between the pair of variables, which is constant and equal to 1/6 whatever the form of the marginal distribution functions of the two variables when knowledge on the pair of variables is the only one available in the two samples. This measure becomes useful in the context of the redu...

In this paper a procedure for measurement error correction based on nonparametric Bayesian networks is proposed. The performance of the proposed method is evaluated using a validation sample collected by Banca d’Italia and a major Italian bank group to investigate the measurement error mechanism in the main financial variables amounts observed in the Banca d’Italia survey on Household Income and Wealth. Specifically, in this paper attention is focused on the bond amounts. By means of Uninet’s programmatic engine working directly from R, data can be corrected unit by unit by sampling from the nonparametric Bayesian network. Thanks to the validation sample, the distances between the true and the imputed values are computed and the procedure is evaluated.

This paper focuses on the estimation of the concentration curve of a finite population, when data are collected according to a complex sampling design with different inclusion probabilities. A (design-based) Hajek type estimator for the Lorenz curve is proposed, and its asymptotic properties are studied. Then, a resampling scheme able to approximate the asymptotic law of the Lorenz curve estimator is constructed. Applications are given to the construction of (i) a confidence band for the Lorenz curve, (ii) confidence intervals for the Gini concentration ratio, and (iii) a test for Lorenz dominance. The merits of the proposed resampling procedure are evaluated through a simulation study.

The aim of the paper is to study the problem of estimating the quantile function of a finite population. Attention is first focused on point estimation, and asymptotic results are obtained. Confidence intervals are then constructed, based on both the following: (i) asymptotic results and (ii) a resampling technique based on rescaling the ‘usual’ bootstrap. A simulation study to compare asymptotic and resampling-based results, as well as an application to a real population, is finally performed.

In this paper Object-Oriented Bayesian networks are proposed as a tool to model measurement errors in a categorical variable due to respondent. A mixed measurement error model is presented and an Object-Oriented Bayesian network implementing such a model is introduced. The insertion of evidence represented by the observed value and its propagation throughout the network yields for each unit the probability distribution of the true value given the observed. Two methods are used to predict the individual true value and their performance is evaluated via simulation.

Among the goals of statistical matching, a very important one is the estimation of the joint distribution of variables not jointly observed in a sample survey but separately available from independent sample surveys. The absence of joint information on the variables of interest leads to uncertainty about the data generating model since the available sample information is unable to discriminate among a set of plausible joint distributions. In the present paper a short review of the concept of uncertainty in statistical matching under logical constraints, as well as how to measure uncertainty for continuous variables is presented. The notion of matching error is related to an appropriate measure of uncertainty and a criterion of selecting matching variables by choosing the variables minimizing such an uncertainty measure is introduced. Finally, a method to choose a plausible joint distribution for the variables of interest via iterative proportional fitting algorithm is described. The proposed methodology is then applied to household income and expenditure data when extra sample information regarding the average propensity to consume is available. This leads to a reconstructed complete dataset where each record includes measures on income and expenditure.

In this paper, a class of resampling techniques for finite populations under $\pi $ps sampling design is introduced. The basic idea on which they rest is a two-step procedure consisting in: (i) constructing a “pseudo-population” on the basis of sample data; (ii) drawing a sample from the predicted population according to an appropriate resampling design. From a logical point of view, this approach is essentially based on the plug-in principle by Efron, at the “sampling design level”. Theoretical justifications based on large sample theory are provided. New approaches to construct pseudo populations based on various forms of calibrations are proposed. Finally, a simulation study is performed.

In this paper the quality of data produced by national statistical institutes and by governmental institutions is considered. In particular, the problem of measurement error is analysed and an integrated Bayesian network decision support system based on non parametric Bayesian networks is proposed for its detection and correction. Non parametric Bayesian networks are graphical models expressing dependence structure via bivariate copulas associated to the edges of the graph. The network structure and the misreport probability are estimated using a validation sample. The Bayesian network model is proposed to decide: i) which records have to be corrected; ii) the kind and amount of correction to be adopted. The proposed correction procedure is applied to the Banca d’Italia Survey on Household Income and Wealth and, specifically, the bond amounts are analysed. Finally, the sensitivity of the conditional distribution of the true value random variable given the observed one to different evidence configurations is studied.

The aim of the paper is to evaluate the matching noise produced by nonparametric imputation techniques referring to the kNN method, both with fixed and variable number of donors k. The matching noise is evaluated formally and via a simulation.

The aim of the paper is to establish asymptotics in sampling finite populations. Asymptotic results are first established for an analogous of the empirical process based on the Hajek estimator of the population distribution function and then extended to Hadamard-differentiable functions. As an application, asymptotic normality of estimated quantiles is provided.

In this paper we propose to use the object-oriented Bayesian networks (OOBNs) architecture to model measurement errors in the Italian survey on household income and wealth (SHIW) 2008 when the variable of interest is categorical. The network is used to stochastically impute microdata for households. Imputation is performed both assuming a misreport probability constant over all the population and learning a Bayesian network for estimating such a probability. Finally, potentialities and possible extensions of this approach are discussed.

In this paper we propose to use the object-oriented Bayesian network (OOBN) architecture to model measurement errors. We then apply our model to the Italian survey on household income and wealth (SHIW) 2008. Attention is focused on errors caused by the respondents. The parameters of the error model are estimated using a validation sample. The network is used to stochastically impute micro data for households. In particular imputation is performed also using an auxiliary variable. Indices are calculated to evaluate the performance of the correction procedure and show that accounting for auxiliary information improves the results. Finally, potentialities and possible extensions of the Bayesian network approach both to the measurement error context and to official statistics problems in general are discussed.

L’obiettivo di questo capitolo e quello di esporre alcuni disegni campionari che possiedono “buone” proprieta, che li rendono particolarmente importanti per le loro potenzialita applicative. Una particolare attenzione sara dedicata agli aspetti algoritmici, in particolare per quel che riguarda la definizione di schemi computazionalmente efficienti per selezionare campioni di unita sulla base di un dato disegno di campionamento (implementazione del disegno campionario). Si tratta di un punto molto importante, in quanto l’uso effettivo di un disegno campionario e legato in modo indissolubile alla possibilita di implementarlo in modo numericamente efficiente, ossia alla possibilita di selezionare in modo numericamente efficiente un campione.

Nella teoria dell’inferenza statistica un ruolo di primo piano e svolto dalla funzione di verosimiglianza. E quindi di un certo interesse studiare se e in che misura il relativo quadro concettuale possa essere adattato al campionamento da popolazioni finite. Prima di iniziare la vera e propria trattazione, e da sottolineare che l’approccio fino ad ora seguito e basato sul disegno, nel senso che: (i) le modalita y i non sono generate da alcun modello; (ii) l’unica fonte di aleatorieta, di incertezza, e quella dovuta alla selezione del campione di unita, la quale e regolata da un disegno (probabilistico) di campionamento.

In questo capitolo ci si occupera principalmente dei problemi di stima della media della popolazione, μ y = Σ i=1 N y i /N. Si accennera inoltre, molto brevemente, anche al problema della stima del totale, θ = Σ i=1 N y i = N μ y A meno che non venga espressamente detto il contrario, nel seguito si assumera sempre che il disegno campionario sia non ordinato e senza ripetizioni. In caso contrario e sufficiente passare alla riduzione del disegno stesso.

In molti casi di interesse la popolazione si puo pensare strutturata in grappoli, i quali potrebbero essere formati da parecchie unita elementari. Se si effettua un campionamento a grappolo, basta campionare pochi grappoli per avere un numero anche molto elevato di unita elementari. Ad esempio, si supponga di voler stimare il tasso di disoccupazione della popolazione di una citta. Le unita elementari di rilevazione sono i singoli individui (diciamo di eta 14–65 anni). In maniera molto semplice, si puo pensare ad essi come raggruppati in grappoli, ciascuno dei quali e costituito da tutti gli individui che vivono in una stessa strada o piazza. Selezionare un campione di strade/piazze e osservare tutti gli individui che vivono negli edifici corrispondenti e un’operazione poco conveniente in quanto, a meno che la numerosita campionaria non sia molto grande, basteranno poche strade/piazze per raggiungere parecchie centinaia o migliaia di individui.

Lo stimatore per quoziente e, come struttura logica, simile a quello per regressione. Poniamoci esattamente nelle condizioni del Capitolo 5, ovvero supponiamo che siano note le modalita x1,..., x N che un carattere ausiliario χ assume in corrispondenza delle unita della popolazione. Indichiamo con μ x = Σ i=1 N x i /N la media del carattere χ nella popolazione. L’obiettivo, come al solito, e quello di stimare la media μ y del carattere γ. In tutto il presente capitolo si assumera che il disegno di campionamento e quello semplice senza ripetizione.

Fino ad ora e sempre stata assunta come data a priori la numerosita campionaria. Ora, in una qualunque rilevazione statistica la numerosita del campione che si utilizza e uno degli elementi piu importanti, per diverse ragioni.

La necessita di informazioni statistiche sempre piu accurate e disponibili in tempi rapidi costituisce indubbiamente uno degli aspetti salienti delle societ a moderne. Tali informazioni sono molto spesso acquisibili solo mediante rilevazioni statistiche, che consistono (almeno in prima approssimazione) nelle attivita di raccolta ed elaborazione di dati statistici riguardanti specifici insiemi di elementi, detti “popolazioni finite”. Gli esempi in proposito sono numerosissimi. Ad es., il nostro obiettivo potrebbe essere quello di ottenere informazioni sulle aziende operanti in uno o piu settori di attivita economica e sul loro fatturato, sugli individui di una data comunita (nazionale, regionale, etc.) e sul loro status lavorativo (occupati, disoccupati, etc.), sulle famiglie residenti in una data area geografica e su aspetti legati alle loro abitudini di consumo e ai loro redditi, sui cittadini aventi diritti politici e sulla loro disponibilita a votare un certo partito politico. Solo se resi disponibili con tempestivita, tali dati possono soddisfare le specifiche esigenze, sia conoscitive che decisionali, di istituzioni pubbliche e organismi privati.

La nozione generale di disegno campionario, cosi come quelle di numerosita e di numerosita effettiva, sono gia state fornite nel Capitolo 2. I disegni di campionamento visti fino ad ora, pur essendo composti da campioni che non hanno tutti necessariamente la stessa probabilita di selezione, hanno una caratteristica comune: sono riconducibili al disegno semplice senza ripetizione (ssr), nel senso che sono sostanzialmente suoi “derivati”. In effetti, e evidente come i disegni di campionamento stratificato, a grappolo, a due stadi semplici, sono essenzialmente varianti del disegno ssr. In questo capitolo e nei successivi, invece, ci si concentrera su disegni campionari non riconducibili a quello semplice, che verranno per brevita denominati disegni a probabilita variabile. Essi presentano importanti problemi di implementazione, in generale di non agevole soluzione. D’altra parte, disegni campionari a probabilita variabile sono ampiamente usati nella pratica applicativa.

Il disegno campionario semplice senza ripetizione (disegno ssr, d’ora in avanti) e probabilmente il piu importante tra tutti i disegni campionari, sia per l’utilizzo diretto che se ne fa, sia perche entra come costituente essenziale di disegni campionari piu complessi, (disegno stratificato, disegno a due stadi, etc.).

Nei disegni campionari finora presi in esame si e sempre assunto che e data una lista di unita elementari (unita di rilevazione) della popolazione, e che la procedura di selezione del campione agisca direttamente su tali unita. Nella terminologia introdotta nella Sezione 1.5, le unita di campionamento coincidono con le unita di osservazione.

A new matching procedure based on imputing missing data by means of a local linear estimator of the underlying population regression function (that is assumed not necessarily linear) is introduced. Such a procedure is compared to other traditional approaches, more precisely hot deck methods as well as methods based on kNN estimators. The relationship between the variables of interest is assumed not necessarily linear. Performance is measured by the matching noise given by the discrepancy between the distribution generating genuine data and the distribution generating imputed values.

In this paper, the difference between the data generating process and the imputation procedures used in statistical matching is evaluated. The investigated imputation procedures are the distance hot deck and those referring to the kNN method, both with fixed and variable number of donors k. The matching noise is evaluated formally and investigated via a simulation.