Search

Studies often seek to estimate the causal effect of a treatment on an outcome using a sample that has been drawn from a larger population. Such samples may not be randomly drawn and researcher may not observe all confounding variables that drive selection into treatment. Though researchers may be content to study causal effects averaged over only the sample in hand, selective sampling and unobserved confounding can still bias such effect estimates, threatening their “internal validity” (Campbell, 1957). Sample selection and unobserved confounding are related in how they threaten internal validity and can be examined in conjunction. We develop graphical tools to help evaluate threats from sample selection and unobserved confounding. These tools also allow us to determine when covariate adjustment can overcome these threats. It will not always be possible to solve these problems, however, and we thus generalize sensitivity analyses for unobserved confounding to also address sample selection. We then consider the use of instrumental variables, which in some cases can be biased by these concerns, but in other cases offer a solution to them. Finally, shifting emphasis to unobserved confounding, we discuss the use of placebo variables in partial identification.

Este trabajo desarrolla un método flexible y semiestructural para cuantificar empíricamente la transmisión no lineal de los shocks de renta a las decisiones de elección de cartera de los hogares, tanto en el margen extensivo como en el intensivo. Modelizo la participación en el mercado de valores y las reglas de asignación de carteras como funciones dependientes de la edad de los componentes persistentes y transitorios de los ingresos, la riqueza y los cambios de gusto no observados. Establezco una identificación no paramétrica y propongo un algoritmo de estimación basado en la simulación, fundado en los recientes desarrollos de la literatura de selección de muestras. Utilizando olas recientes de datos de la PSID, observo efectos heterogéneos sobre la renta y la riqueza en los márgenes extensivo e intensivo, en las dimensiones de riqueza y ciclo vital. Estos resultados sugieren que las preferencias son heterogéneas a lo largo de la distribución de la riqueza y del ciclo vital. Además, en los ejercicios de respuesta al impulso, encuentro respuestas considerables del margen extensivo a las perturbaciones persistentes de la renta. Por último, advierto heterogeneidad en los costes de participación de los hogares en la distribución de la riqueza., This paper develops a flexible, semi-structural framework to empirically quantify the non-linear transmission of income shocks to household portfolio choice decisions both at the extensive and intensive margins. I model stock market participation and portfolio allocation rules as age-dependent functions of persistent and transitory earnings components, wealth and unobserved taste shifters. I establish non-parametric identification and propose a tractable, simulation-based estimation algorithm, building on recent developments in the sample selection literature. Using recent waves of PSID data, I find heterogeneous income and wealth effects on both extensive and intensive margins, over the wealth and life-cycle dimensions. These results suggest that preferences are heterogeneous across the wealth distribution and over the life cycle. Moreover, in impulse response exercises, I find sizeable extensive margin responses to persistent income shocks. Finally, I find heterogeneity in participation costs across households in the wealth distribution.

SARS-CoV-2 vaccines give rise to positive externalities on population health, society and the economy in addition to protecting the health of vaccinated individuals. Hence, the social value of such a vaccine exceeds its market value. This paper estimates the willingness to pay (WTP) for a hypothetical SARS-CoV-2 vaccine (or shadow prices), in four countries, namely the United States (US), the United Kingdom, Spain and Italy during the first wave of the pandemic when COVID-19 vaccines were in development but not yet approved. WTP estimates are elicited using a payment card method to avoid "yea saying" biases, and we study the effect of protest responses, sample selection bias, as well as the influence of trust in government and risk exposure when estimating the WTP. Our estimates suggest evidence of an average value of a hypothetical vaccine of 100-200 US dollars once adjusted for purchasing power parity. Estimates are robust to a number of checks.

Monitoring and updating calibration models are common tasks when analytical methods are based on near-infrared spectroscopy. This work describes a situation in which a PLS calibration model that is used routinely for the determination of phosphorus content in pig faeces in digestibility studies had to be updated in order to be used with the faeces collected in a new trial with phytases. An approach based on D-optimality is presented that selects a reduced number of the new samples to be analyzed with the reference analytical method so that the small set is used to confirm the need to update the model and validate it. The rest of the new samples that had not been selected by the algorithm were accurately predicted with the updated model. The updated model maintained its previous performance for the samples in the validation set (an RMSEP of 1.58 g kg−1 compared with an RMSEP of 1.54 g kg−1 before the update) and the prediction error for the new samples was RMSECV = 1.95 g kg−1, much lower than the RMSEP = 11.38 g kg−1 obtained before the model update. In addition, the predictive ability of the updated PLS model was significantly better than updated models selecting the reduced dataset using other sample selection methods such as Kennard-Stone, a leverage-based selection method and random selection.

As more devices with different service requirements become connected to networked systems, such as Internet of Things (IoT) devices, maintaining quality of service becomes increasingly difficult. Large data sets can be obtained ahead of time in networks to train prediction models offline, however, resulting in high computational costs. Online learning is an alternative approach where a smaller cache of fixed size is maintained for training using sample selection algorithms, allowing for lower computational costs and real-time model re-computation. This project has resulted in two newly designed sample selection algorithms, Binned Relevance and Redundancy Sample Selection (BRR-SS) and Autoregressive First, In First Out-buffer (AR-FIFO). The algorithms are evaluated on data traces retrieved from a Key Value store and a Video on Demand service. Prediction accuracy of the resulting model while using the sample selection algorithms and the time to process a received sample is evaluated and compared to the pre-existing Reservoir Sampling (RS) and Relevance and Redundancy Sample Selection (RR-SS) with and without model re-computation. The results show that, while RS maintains the lowest computational overhead, BRR-SS outperforms both RS and RR-SS in prediction accuracy on the investigated traces. AR-FIFO, with its low computational cost, outperforms offline learning for larger cache sizes on the Key Value data set but shows inconsistencies on the Video on Demand trace. Model re-computation results in reduced error rates and significantly lowered variance on the investigated data traces, where periodic model re-computation overall outperforms change detection in practicality, prediction accuracy, and computational overhead., Allteftersom fler enheter med olika servicekrav ansluts till nätverkssystem, såsom Internet of Things (IoT) enheter, ökar svårigheten att erhålla nödvändig servicekvalitet. Nätverk kan ge upphov till stora datamängder för träning av prediktionsmodeller offline, dock till en hög beräkningskostnad. Ett alternativt tillvägagångssätt är onlineinlärning där en mindre cache av fast storlek upprätthålls för träning med hjälp av datapunkturvalsalgoritmer. Detta möjliggör lägre beräkningskostnader samt realtidsmodellomräkningar. Detta projekt har resulterat i två nydesignade datapunkturvalsalgoritmer, Binned Relevance and Redundancy Sample Selection (BRR-SS) och Autoregressive First In, First Out-buffer (AR-FIFO). Algoritmerna utvärderas på dataspår som hämtats från ett Key Value-lager och en Video on Demand-tjänst. Förutsägelseförmåga för den resulterande modellen när datapunkturvalsalgoritmerna används och tid för bearbetning av mottagen datapunkt utvärderas och jämförs med dem redan existerande Reservoir Sampling (RS) och Relevance and Redundancy Sample Selection (RR-SS), med och utan modellomräkning. RS resulterar i lägst beräkningskostnad medan BRR-SS överträffar både RS och RR-SS i förutsägelseförmåga på dem undersökta spåren. AR-FIFO, med sin låga beräkningskostnad, överträffar offlineinlärning för större cachestorlekar på Key Value-spåret, men visar inkonsekvent beteende på Video on Demand-spåret. Modellomräkning resulterar i mindre fel och avsevärt sänkt varians på dem undersökta spåren, där periodisk modellomräkning totalt sett överträffar förändringsdetektering i praktikalitet, förutsägelseförmåga och beräkningskostnad., Kandidatexjobb i elektroteknik 2022, KTH, Stockholm

As more devices with different service requirements become connected to networked systems, such as Internet of Things (IoT) devices, maintaining quality of service becomes increasingly difficult. Large data sets can be obtained ahead of time in networks to train prediction models offline, however, resulting in high computational costs. Online learning is an alternative approach where a smaller cache of fixed size is maintained for training using sample selection algorithms, allowing for lower computational costs and real-time model re-computation. This project has resulted in two newly designed sample selection algorithms, Binned Relevance and Redundancy Sample Selection (BRR-SS) and Autoregressive First, In First Out-buffer (AR-FIFO). The algorithms are evaluated on data traces retrieved from a Key Value store and a Video on Demand service. Prediction accuracy of the resulting model while using the sample selection algorithms and the time to process a received sample is evaluated and compared to the pre-existing Reservoir Sampling (RS) and Relevance and Redundancy Sample Selection (RR-SS) with and without model re-computation. The results show that, while RS maintains the lowest computational overhead, BRR-SS outperforms both RS and RR-SS in prediction accuracy on the investigated traces. AR-FIFO, with its low computational cost, outperforms offline learning for larger cache sizes on the Key Value data set but shows inconsistencies on the Video on Demand trace. Model re-computation results in reduced error rates and significantly lowered variance on the investigated data traces, where periodic model re-computation overall outperforms change detection in practicality, prediction accuracy, and computational overhead., Allteftersom fler enheter med olika servicekrav ansluts till nätverkssystem, såsom Internet of Things (IoT) enheter, ökar svårigheten att erhålla nödvändig servicekvalitet. Nätverk kan ge upphov till stora datamängder för träning av prediktionsmodeller offline, dock till en hög beräkningskostnad. Ett alternativt tillvägagångssätt är onlineinlärning där en mindre cache av fast storlek upprätthålls för träning med hjälp av datapunkturvalsalgoritmer. Detta möjliggör lägre beräkningskostnader samt realtidsmodellomräkningar. Detta projekt har resulterat i två nydesignade datapunkturvalsalgoritmer, Binned Relevance and Redundancy Sample Selection (BRR-SS) och Autoregressive First In, First Out-buffer (AR-FIFO). Algoritmerna utvärderas på dataspår som hämtats från ett Key Value-lager och en Video on Demand-tjänst. Förutsägelseförmåga för den resulterande modellen när datapunkturvalsalgoritmerna används och tid för bearbetning av mottagen datapunkt utvärderas och jämförs med dem redan existerande Reservoir Sampling (RS) och Relevance and Redundancy Sample Selection (RR-SS), med och utan modellomräkning. RS resulterar i lägst beräkningskostnad medan BRR-SS överträffar både RS och RR-SS i förutsägelseförmåga på dem undersökta spåren. AR-FIFO, med sin låga beräkningskostnad, överträffar offlineinlärning för större cachestorlekar på Key Value-spåret, men visar inkonsekvent beteende på Video on Demand-spåret. Modellomräkning resulterar i mindre fel och avsevärt sänkt varians på dem undersökta spåren, där periodisk modellomräkning totalt sett överträffar förändringsdetektering i praktikalitet, förutsägelseförmåga och beräkningskostnad., Kandidatexjobb i elektroteknik 2022, KTH, Stockholm

This paper empirically analyses the relationship between urban form and Urban Heat Island (UHI) in a dataset of 523 European cities that matches remotely sensed land-use and surface temperature data. A UHI anomaly is defined as an uninterrupted streak of days where the temperature differential measured at 12.00 AM between the city core and its surroundings is higher than a given threshold. From this definition, three UHI indicators are obtained: mean intensity, mean duration of the event and occurrence rate. We study the influence of urban morphology on the UHI indictors with a Heckman model. A sample selection bias is detected for mean intensity and mean duration. The estimation results also show that some urban morphological features have a mitigating effect, while some others play a role at the adaptation level.

[ES] El objetivo que se persigue en este trabajo es explicar las técnicas estadísticas que se utilizan para facilitar el análisis de la información a la hora de realizar una auditoría. Este análisis conlleva manejar altos volúmenes de datos, por lo que es necesario emplear métodos estadísticos para seleccionar aquella parte de información que se va a analizar. En el presente trabajo se va a desarrollar tanto de manera teórica como practica el uso de la estadística en la ejecución de trabajos de auditoria, concretamente en la técnica de la circularización de clientes y proveedores. Se explican los métodos por los que el auditor realiza la selección de los clientes y proveedores a circularizar, entre ellos se encuentra el método estadístico de la unidad monetaria. Se detallará el proceso que se sigue en la aplicación de este método. Finalmente, con las conclusiones que se hayan extraído, se hablará de las distintas opiniones que pueden ofrecer los informes de auditoría., [EN] The objective pursued in this work is to explain the statistical techniques that are used to facilitate the analysis of information when conducting an audit. This analysis involves handling high volumes of data, so it is necessary to use statistical methods to select that part of the information to be analyzed. In this paper, the use of statistics in the execution of audit work will be developed both in a theoretical and practical way, specifically in the technique of circularization of clients and suppliers. The methods by which the auditor selects the clients and suppliers to be circulated are explained, among them is the statistical method of the monetary unit. The process followed in the application of this method will be detailed. Finally, with the conclusions that have been drawn, we will discuss the different opinions that the audit reports can offer., [CA] L'objectiu que es persegueix en aquest treball és explicar les tècniques estadístiques que s'utilitzen per facilitar l'anàlisi de la informació a l'hora de realitzar una auditoria. Aquesta anàlisi comporta gestionar alts volums de dades, de manera que cal emprar mètodes estadístics per seleccionar aquella part d'informació que es va a analitzar. En el present treball es va a desenvolupar tant de manera teòrica com practica l'ús de l'estadística en l'execució de treballs d'auditoria, concretament en la tècnica de la circularització de clients. S'expliquen els mètodes pels quals l'auditor realitza la selecció dels clients a circularitzar, entre ells es troba el mètode estadístic de la unitat monetària. Es detallarà el procés que se segueix en l'aplicació d'aquest mètode. Finalment, amb les conclusions que s'hagin extret, es parlarà de les diferents opinions que poden oferir els informes d'auditoria.

[ES] El objetivo que se persigue en este trabajo es explicar las técnicas estadísticas que se utilizan para facilitar el análisis de la información a la hora de realizar una auditoría. Este análisis conlleva manejar altos volúmenes de datos, por lo que es necesario emplear métodos estadísticos para seleccionar aquella parte de información que se va a analizar. En el presente trabajo se va a desarrollar tanto de manera teórica como practica el uso de la estadística en la ejecución de trabajos de auditoria, concretamente en la técnica de la circularización de clientes y proveedores. Se explican los métodos por los que el auditor realiza la selección de los clientes y proveedores a circularizar, entre ellos se encuentra el método estadístico de la unidad monetaria. Se detallará el proceso que se sigue en la aplicación de este método. Finalmente, con las conclusiones que se hayan extraído, se hablará de las distintas opiniones que pueden ofrecer los informes de auditoría., [EN] The objective pursued in this work is to explain the statistical techniques that are used to facilitate the analysis of information when conducting an audit. This analysis involves handling high volumes of data, so it is necessary to use statistical methods to select that part of the information to be analyzed. In this paper, the use of statistics in the execution of audit work will be developed both in a theoretical and practical way, specifically in the technique of circularization of clients and suppliers. The methods by which the auditor selects the clients and suppliers to be circulated are explained, among them is the statistical method of the monetary unit. The process followed in the application of this method will be detailed. Finally, with the conclusions that have been drawn, we will discuss the different opinions that the audit reports can offer., [CA] L'objectiu que es persegueix en aquest treball és explicar les tècniques estadístiques que s'utilitzen per facilitar l'anàlisi de la informació a l'hora de realitzar una auditoria. Aquesta anàlisi comporta gestionar alts volums de dades, de manera que cal emprar mètodes estadístics per seleccionar aquella part d'informació que es va a analitzar. En el present treball es va a desenvolupar tant de manera teòrica com practica l'ús de l'estadística en l'execució de treballs d'auditoria, concretament en la tècnica de la circularització de clients. S'expliquen els mètodes pels quals l'auditor realitza la selecció dels clients a circularitzar, entre ells es troba el mètode estadístic de la unitat monetària. Es detallarà el procés que se segueix en l'aplicació d'aquest mètode. Finalment, amb les conclusions que s'hagin extret, es parlarà de les diferents opinions que poden oferir els informes d'auditoria.

Developing chemometric models from near-infrared (NIR) spectra requires the use of a representative calibration set of the entire population. Therefore, generally, the calibration procedure requires a large number of resources. For that reason, there is a great interest in identifying the most spectrally representative samples within a large population set. In this study, principal component and hierarchical clustering analyses have been compared for their ability to provide different representative calibration sets. The calibration sets generated have been used to control the technological maturity of grapes and total phenolic compounds of grape skins in red and white cultivars. Finally, the accuracy and precision of the models obtained with these calibration sets resulted from the application of the selection algorithms studied have been compared with each other and with the whole set of samples using an external validation set. Most of the standard errors of prediction (SEP) in external validation obtained from the reduced data sets were not significantly different from those obtained using the whole data set. Moreover, sample subsets resulting from hierarchical clustering analysis appear to produce slightly better results.

[ES] El objetivo que se persigue en este trabajo es explicar las técnicas estadísticas que se utilizan para facilitar el análisis de la información a la hora de realizar una auditoría. Este análisis conlleva manejar altos volúmenes de datos, por lo que es necesario emplear métodos estadísticos para seleccionar aquella parte de información que se va a analizar. En el presente trabajo se va a desarrollar tanto de manera teórica como practica el uso de la estadística en la ejecución de trabajos de auditoria, concretamente en la técnica de la circularización de clientes y proveedores. Se explican los métodos por los que el auditor realiza la selección de los clientes y proveedores a circularizar, entre ellos se encuentra el método estadístico de la unidad monetaria. Se detallará el proceso que se sigue en la aplicación de este método. Finalmente, con las conclusiones que se hayan extraído, se hablará de las distintas opiniones que pueden ofrecer los informes de auditoría., [EN] The objective pursued in this work is to explain the statistical techniques that are used to facilitate the analysis of information when conducting an audit. This analysis involves handling high volumes of data, so it is necessary to use statistical methods to select that part of the information to be analyzed. In this paper, the use of statistics in the execution of audit work will be developed both in a theoretical and practical way, specifically in the technique of circularization of clients and suppliers. The methods by which the auditor selects the clients and suppliers to be circulated are explained, among them is the statistical method of the monetary unit. The process followed in the application of this method will be detailed. Finally, with the conclusions that have been drawn, we will discuss the different opinions that the audit reports can offer., [CA] L'objectiu que es persegueix en aquest treball és explicar les tècniques estadístiques que s'utilitzen per facilitar l'anàlisi de la informació a l'hora de realitzar una auditoria. Aquesta anàlisi comporta gestionar alts volums de dades, de manera que cal emprar mètodes estadístics per seleccionar aquella part d'informació que es va a analitzar. En el present treball es va a desenvolupar tant de manera teòrica com practica l'ús de l'estadística en l'execució de treballs d'auditoria, concretament en la tècnica de la circularització de clients. S'expliquen els mètodes pels quals l'auditor realitza la selecció dels clients a circularitzar, entre ells es troba el mètode estadístic de la unitat monetària. Es detallarà el procés que se segueix en l'aplicació d'aquest mètode. Finalment, amb les conclusions que s'hagin extret, es parlarà de les diferents opinions que poden oferir els informes d'auditoria.

Developing chemometric models from near-infrared (NIR) spectra requires the use of a representative calibration set of the entire population. Therefore, generally, the calibration procedure requires a large number of resources. For that reason, there is a great interest in identifying the most spectrally representative samples within a large population set. In this study, principal component and hierarchical clustering analyses have been compared for their ability to provide different representative calibration sets. The calibration sets generated have been used to control the technological maturity of grapes and total phenolic compounds of grape skins in red and white cultivars. Finally, the accuracy and precision of the models obtained with these calibration sets resulted from the application of the selection algorithms studied have been compared with each other and with the whole set of samples using an external validation set. Most of the standard errors of prediction (SEP) in external validation obtained from the reduced data sets were not significantly different from those obtained using the whole data set. Moreover, sample subsets resulting from hierarchical clustering analysis appear to produce slightly better results.

In data-driven network and systems engineering, we often train models offline using measurement data collected from networks. Offline learning achieves good results but has drawbacks. For example, model training incurs a high computational cost and the training process takes a long time. In this project, we follow an online approach for model training. The approach involves a cache of fixed size to store measurement samples and recomputation of ML models based on the current cache. Key to this approach are sample selection algorithms that decide which samples are stored in the cache and which are evicted. We implement three sample selection methods in this project: reservoir sampling, maximum entropy sampling and maximum coverage sampling. In the context of sample selection methods, we evaluate model recomputation methods to control when to retrain the model using the samples in the current cache and use the retrained model to predict the following samples before the next recomputation moment. We compare three recomputation strategies: no recomputation, periodic recomputation and recomputation using the ADWIN algorithm. We evaluate the three sample selection methods on five datasets. One of them is the FedCSIS 2020 Challenge dataset and the other four are KTH testbed datasets. We find that maximum entropy sampling can achieve quite good performance compared to other sample selection methods and that recomputation using the ADWIN algorithm can help reduce the number of recomputations and does not affect the prediction performance., Vid utveckling och underhåll av datornätverk och system så används ofta maskininlärningsmodeller (ML) som beräknats offline med datavärden som insamlats från nätverket. Att beräkna ML-modeller offline ger bra resultat men har nackdelar. Beräkning av ML-modeller är tidskrävande och medför en hög beräkningskostnad. I detta projekt undersöker vi en metod för att beräkna ML-modeller online. Metoden använder en cache av fixerad storlek för att lagra mätningsvärden och omberäknar ML-modeller baserat på innehållet i cachen. Nyckeln till denna metod är användandet av urvalsalgoritmer som avgör vilka mätningsvärden som ska lagras i cachen och vilka som ska tas bort. Vi tillämpar tre urvalsmetoder: urval baserat på en behållare av fixerad storlek, urval baserat på maximal entropi, samt urval baserat på maximal täckning. Vid användning av urvalsmetoder så utvärderar vi metoder för att avgöra när en ML-modell ska omberäknas baserat på urvalet i cachen. Den omberäknade ML-modellen används sedan för att göra prediktioner tills dess att modellen omberäknas igen. Vi utvärderar tre strategier för att avgöra när en modell ska omberäknas: ingen omberäkning, periodisk omberäkning, samt omberäkning baserat på ADWIN-algoritmen. Vi utvärderar tre urvalsmetoder på fem olika datauppsättningar. En av datauppsättningarna är baserat på FedCSIS 2020 Challenge och de andra fyra datauppsättningarna har insamlats från en testbädd på KTH. Vi _nner att urval baserat på maximal entropi uppnår bra prestanda jämfört med de andra urvalsmetoderna samt att en omberäkningstrategi baserat på ADWIN-algoritmen kan minska antalet omberäkningar och försämrar inte prediktionsprestandan.

Estimating true treatment effects in the presence of selection bias is a common problem in program evaluation, but it is uncommon to quantify the share of a total apparent effect that is “true”. We identify multiple methods for doing so, in the context of isolating the true causal influence of the built environment on travel behavior in the presence of residential self-selection (RSS). Among the different approaches for dealing with self-selection, we use three: statistical control (SC) modeling, propensity score-based techniques (PS), and sample selection (SS) modeling. Our objective is to identify multiple ways to estimate the proportion of the total apparent effect of the built environment on travel behavior that is due to the built environment itself, which we call the “built environment proportion”, or BEP. We present and evaluate three categories of methods for estimating this measure (each native to one of the approaches): variance explained, modular effects, and treatment effects. A BEP formula associated with a given method can be applied to its native approach or cross-applied to other approaches. We identify and enumerate 47 potential BEPs. The methods presented here can be applied in other contexts involving treatment evaluation in the presence of selection bias.

Developing chemometric models from near-infrared (NIR) spectra requires the use of a representative calibration set of the entire population. Therefore, generally, the calibration procedure requires a large number of resources. For that reason, there is a great interest in identifying the most spectrally representative samples within a large population set. In this study, principal component and hierarchical clustering analyses have been compared for their ability to provide different representative calibration sets. The calibration sets generated have been used to control the technological maturity of grapes and total phenolic compounds of grape skins in red and white cultivars. Finally, the accuracy and precision of the models obtained with these calibration sets resulted from the application of the selection algorithms studied have been compared with each other and with the whole set of samples using an external validation set. Most of the standard errors of prediction (SEP) in external validation obtained from the reduced data sets were not significantly different from those obtained using the whole data set. Moreover, sample subsets resulting from hierarchical clustering analysis appear to produce slightly better results.

Data-driven functions for network operation and management are based upon AI/ML methods whose models are usually trained offline with measurement data collected through monitoring. Online learning provides an alternative with the prospect of shorter learning times and lower overhead, suitable for edge or other resource-constraint environments. We propose an approach to online learning that involves a cache of fixed size to store measurement samples and periodic re-computation of ML models. Key to this approach are sample selection algorithms that decide which samples are stored in the cache and which are evicted. We present and evaluate four sample selection algorithms, all of which are derived from well-studied algorithms, and we specifically argue that feature selection algorithms can be used for our purpose. We perform an extensive evaluation of these algorithms for the task of performance prediction using data from an in-house testbed. We find that one of them (RR-SS) leads to models that achieve a prediction accuracy close to that obtained through offline learning, but at a much lower cost., Part of proceedings: ISBN 978-3-903176-32-4, QC 20230117

Consuming too much fat, sugar, and salt is associated with adverse health outcomes. Food reformulation is one possible strategy to enhance the food environment by improving the nutritional quality of commercial products. However, food reformulation faces many hindrances. One way to alleviate some of these hindrances is to embrace a multicriteria approach that is based on a market inventory. In this objective, additional sensory screening and water content analyses allow going beyond nutrition and composition information on the packaging. However, due to feasibility reasons for later in-depth analyses, it is necessary to work with several reduced and manageable products. To the best of the authors' knowledge, in the literature, there is no sample selection approach taking into account multiple criteria as a base for future food reformulation. The overall aim of this paper is to propose a method to select the best representative products from the market base, for future reformulation by going beyond nutrition and composition information on the packaging. This approach considered therefore nutrition, composition, economic, water content, and sensory information with the example of the cookies market. The first step is the creation of an extensive cookie database including sensory and water content information. In total 178 cookies among the French market were identified, then focus was placed on 62 chocolate chip cookies only. Sensory screening and water content analyses of all 62 products were conducted. The second step is to make an informed subset selection, thanks to a cluster analysis based on 11 nutrition, composition, and water content variables. A representative subset of 18 cookies could be derived from the obtained clusters. The representativity was evaluated with statistical uni- and multivariate analyses. Results showed a broad variety of chocolate chips cookies with a large nutritional, compositional, water content, and sensory differences. These results h

In data-driven network and systems engineering, we often train models offline using measurement data collected from networks. Offline learning achieves good results but has drawbacks. For example, model training incurs a high computational cost and the training process takes a long time. In this project, we follow an online approach for model training. The approach involves a cache of fixed size to store measurement samples and recomputation of ML models based on the current cache. Key to this approach are sample selection algorithms that decide which samples are stored in the cache and which are evicted. We implement three sample selection methods in this project: reservoir sampling, maximum entropy sampling and maximum coverage sampling. In the context of sample selection methods, we evaluate model recomputation methods to control when to retrain the model using the samples in the current cache and use the retrained model to predict the following samples before the next recomputation moment. We compare three recomputation strategies: no recomputation, periodic recomputation and recomputation using the ADWIN algorithm. We evaluate the three sample selection methods on five datasets. One of them is the FedCSIS 2020 Challenge dataset and the other four are KTH testbed datasets. We find that maximum entropy sampling can achieve quite good performance compared to other sample selection methods and that recomputation using the ADWIN algorithm can help reduce the number of recomputations and does not affect the prediction performance., Vid utveckling och underhåll av datornätverk och system så används ofta maskininlärningsmodeller (ML) som beräknats offline med datavärden som insamlats från nätverket. Att beräkna ML-modeller offline ger bra resultat men har nackdelar. Beräkning av ML-modeller är tidskrävande och medför en hög beräkningskostnad. I detta projekt undersöker vi en metod för att beräkna ML-modeller online. Metoden använder en cache av fixerad storlek för att lagra mätningsvärden och omberäknar ML-modeller baserat på innehållet i cachen. Nyckeln till denna metod är användandet av urvalsalgoritmer som avgör vilka mätningsvärden som ska lagras i cachen och vilka som ska tas bort. Vi tillämpar tre urvalsmetoder: urval baserat på en behållare av fixerad storlek, urval baserat på maximal entropi, samt urval baserat på maximal täckning. Vid användning av urvalsmetoder så utvärderar vi metoder för att avgöra när en ML-modell ska omberäknas baserat på urvalet i cachen. Den omberäknade ML-modellen används sedan för att göra prediktioner tills dess att modellen omberäknas igen. Vi utvärderar tre strategier för att avgöra när en modell ska omberäknas: ingen omberäkning, periodisk omberäkning, samt omberäkning baserat på ADWIN-algoritmen. Vi utvärderar tre urvalsmetoder på fem olika datauppsättningar. En av datauppsättningarna är baserat på FedCSIS 2020 Challenge och de andra fyra datauppsättningarna har insamlats från en testbädd på KTH. Vi _nner att urval baserat på maximal entropi uppnår bra prestanda jämfört med de andra urvalsmetoderna samt att en omberäkningstrategi baserat på ADWIN-algoritmen kan minska antalet omberäkningar och försämrar inte prediktionsprestandan.

Developing chemometric models from near-infrared (NIR) spectra requires the use of a representative calibration set of the entire population. Therefore, generally, the calibration procedure requires a large number of resources. For that reason, there is a great interest in identifying the most spectrally representative samples within a large population set. In this study, principal component and hierarchical clustering analyses have been compared for their ability to provide different representative calibration sets. The calibration sets generated have been used to control the technological maturity of grapes and total phenolic compounds of grape skins in red and white cultivars. Finally, the accuracy and precision of the models obtained with these calibration sets resulted from the application of the selection algorithms studied have been compared with each other and with the whole set of samples using an external validation set. Most of the standard errors of prediction (SEP) in external validation obtained from the reduced data sets were not significantly different from those obtained using the whole data set. Moreover, sample subsets resulting from hierarchical clustering analysis appear to produce slightly better results.

Developing chemometric models from near-infrared (NIR) spectra requires the use of a representative calibration set of the entire population. Therefore, generally, the calibration procedure requires a large number of resources. For that reason, there is a great interest in identifying the most spectrally representative samples within a large population set. In this study, principal component and hierarchical clustering analyses have been compared for their ability to provide different representative calibration sets. The calibration sets generated have been used to control the technological maturity of grapes and total phenolic compounds of grape skins in red and white cultivars. Finally, the accuracy and precision of the models obtained with these calibration sets resulted from the application of the selection algorithms studied have been compared with each other and with the whole set of samples using an external validation set. Most of the standard errors of prediction (SEP) in external validation obtained from the reduced data sets were not significantly different from those obtained using the whole data set. Moreover, sample subsets resulting from hierarchical clustering analysis appear to produce slightly better results.

This paper compares three different word image representations as base for label free sample selection for word spotting in historical handwritten documents. These representations are a temporal pyramid representation based on pixel counts, a graph based representation, and a pyramidal histogram of characters (PHOC) representation predicted by a PHOCNet trained on synthetic data. We show that the PHOC representation can help to reduce the amount of required training samples by up to 69% depending on the dataset, if it is learned iteratively in an active learning like fashion. While this works for larger datasets containing about 1 700 images, for smaller datasets with 100 images, we find that the temporal pyramid and the graph representation perform better., open access

Various techniques have been applied to address residential self-selection (RSS) bias in estimating the impact of the built environment on travel behavior. We investigate how much the diverse results obtained are due to the method used, focusing on the most-often applied approaches of statistical control modeling, propensity score-based techniques, and sample selection modeling. First, a companion paper identifies 47 plausible ways to estimate a key quantity of interest, namely, the proportion of the total apparent effect of the built environment on travel behavior that is due to the built environment itself, which we call the “built environment proportion”, or BEP. Using a single sample, the present paper provides empirical estimates of 28 of the 47 BEP formulas, and creates and applies a framework for comparing those BEPs. We find that BEPs based on modular effects or treatment effects involving probability weights, and those applied to two-equation models, are among the ten best (based on the goodness of fit of the underlying models and on various measures of quality of the BEP). For our dataset, the ten best BEP averages fall between 0.506 and 0.772, with an overall average of 0.617. Thus, our best estimate is that for this application, about 38% of the total apparent effect of the built environment is due to RSS. However, the range of these averages indicates that even when controlling for as many confounding factors as possible, and even when using the best methods for computing the BEP, we still find substantial variation in the answer to the question: how much of the built environment's apparent total influence on travel behavior is due to self-selection?

This dissertation consists of three chapters that study unobserved heterogeneity in panel and network data models. In Chapter 1, I propose a semi-nonparametric panel data model with a latent group structure. I assume that individual parameters are heterogeneous across groups but homogeneous within a group while the group membership is unknown. I first approximate the infinite-dimensional function with a sieve expansion; then, I propose a Classifier-Lasso(C-Lasso) procedure to simultaneously identify the individuals’ membership and estimate the group-specific parameters. I show that: (i) the classification exhibits uniform consistency; (ii) C-Lasso and post-Lasso estimators achieve oracle properties so that they are asymptotically equivalent to infeasible estimators as if the group membership is known; and (iii) the estimators are consistent and asymptotically normally distributed. Simulations demonstrate an excellent finite sample performance of this approach in both classification and estimation.In Chapter 2 (joint with Wenyu Zhou), we study a nonparametric additive panel regression model with grouped heterogeneity. The model can be regarded as a natural extension to the heterogeneous panel model studied in Su, Shi, and Phillips (2016). We propose to estimate the nonparametric components using a sieve-approximation-based Classifier-Lasso method. We establish the asymptotic properties of the estimator and show that they enjoy the so-called oracle property. In addition, we present the decision rule for group classification and establish its consistency. Then, a BIC-type information criterion is developed to determine the group pattern of each nonparametric component. We further investigate the finite sample performance of the estimation method and the information criterion through Monte Carlo simulations. Results show that both work well. Finally, we apply the model and the estimation method to study the demand for cigarettes in the United States using panel data of

This paper studies the evolution of the gender wage gap in Colombia at different points of wage distributions. Using DiNardo, Fortin, and Lemieux's (1996) decomposition, we find that the enormous increase in female workers' educational achievement has helped to reduce the gender gap, mainly at the top of the wage distribution. However, this effect has been countered by the reduction in the proportion of female workers in the public sector and those with indefinite contracts. Moreover, using the Arellano et al. (2017) methodology, we estimate the adjusted gender wage gap whilst controlling for sample selection. In both analysed years, our main finding was a clear glass-ceiling pattern that was slightly reduced in 2010.

This paper studies the evolution of the gender wage gap in Colombia at different points of wage distributions. Using DiNardo, Fortin, and Lemieux’s (1996) decomposition, we find that the enormous increase in female workers’ educational achievement has helped to reduce the gender gap, mainly at the top of the wage distribution. However, this effect has been countered by the reduction in the proportion of female workers in the public sector and those with indefinite contracts. Moreover, using the Arellano et al. (2017) methodology, we estimate the adjusted gender wage gap whilst controlling for sample selection. In both analysed years, our main finding was a clear glass-ceiling pattern that was slightly reduced in 2010., Este artículo estudia la brecha salarial por género en Colombia en diferentes puntos de la distribución de salarios. Con la descomposición de DiNardo et al. (1996) se encuentra que el enorme incremento en el logro educacional de las trabajadoras femeninas ha ayudado a reducir la brecha de género, principalmente en la cola alta de la distribución. Sin embargo, este efecto ha sido contrarrestado por la reducción en la proporción de mujeres en el sector público y en aquellas con contrato indefinido. Además, con la metodología de Arellano et al. (2016) se estima la brecha de género ajustada controlando por sesgo de selección. Se encuentra un patrón de techo de cristal en ambos años analizados, que se ha reducido ligeramente en el 2010., Este artigo estuda a brecha salarial por gênero na Colômbia em diferentes pontos da distribuição de salários. Com a decomposição de DiNardo et al. (1996), percebe-se que o enorme aumento nos objetivos educacionais das trabalhadoras femininas ajudou na redução da brecha de gênero, principalmente na alta da distribuição. Contudo, este efeito tem sido combatido pela redução na proporção de mulheres no setor público naquelas com contrato indefinido. Além do mais, com a metodologia de Arellano et al. (2016), estima-se uma brecha de gênero ajustada controlada por meio de viés de seleção. Percebe-se também um padrão de teto de cristal em ambos os anos analisados, com uma ligeira redução em 2010.

This paper studies the evolution of the gender wage gap in Colombia at different points of wage distributions. Using DiNardo, Fortin, and Lemieux's (1996) decomposition, we find that the enormous increase in female workers' educational achievement has helped to reduce the gender gap, mainly at the top of the wage distribution. However, this effect has been countered by the reduction in the proportion of female workers in the public sector and those with indefinite contracts. Moreover, using the Arellano et al. (2017) methodology, we estimate the adjusted gender wage gap whilst controlling for sample selection. In both analysed years, our main finding was a clear glass-ceiling pattern that was slightly reduced in 2010.

This paper studies the evolution of the gender wage gap in Colombia at different points of wage distributions. Using DiNardo, Fortin, and Lemieux’s (1996) decomposition, we find that the enormous increase in female workers’ educational achievement has helped to reduce the gender gap, mainly at the top of the wage distribution. However, this effect has been countered by the reduction in the proportion of female workers in the public sector and those with indefinite contracts. Moreover, using the Arellano et al. (2017) methodology, we estimate the adjusted gender wage gap whilst controlling for sample selection. In both analysed years, our main finding was a clear glass-ceiling pattern that was slightly reduced in 2010., Este artigo estuda a brecha salarial por gênero na Colômbia em diferentes pontos da distribuição de salários. Com a decomposição de DiNardo et al. (1996), percebe-se que o enorme aumento nos objetivos educacionais das trabalhadoras femininas ajudou na redução da brecha de gênero, principalmente na alta da distribuição. Contudo, este efeito tem sido combatido pela redução na proporção de mulheres no setor público naquelas com contrato indefinido. Além do mais, com a metodologia de Arellano et al. (2016), estima-se uma brecha de gênero ajustada controlada por meio de viés de seleção. Percebe-se também um padrão de teto de cristal em ambos os anos analisados, com uma ligeira redução em 2010., Este artículo estudia la brecha salarial por género en Colombia en diferentes puntos de la distribución de salarios. Con la descomposición de DiNardo et al. (1996) se encuentra que el enorme incremento en el logro educacional de las trabajadoras femeninas ha ayudado a reducir la brecha de género, principalmente en la cola alta de la distribución. Sin embargo, este efecto ha sido contrarrestado por la reducción en la proporción de mujeres en el sector público y en aquellas con contrato indefinido. Además, con la metodología de Arellano et al. (2016) se estima la brecha de género ajustada controlando por sesgo de selección. Se encuentra un patrón de techo de cristal en ambos años analizados, que se ha reducido ligeramente en el 2010.

This dissertation consists of three chapters that study unobserved heterogeneity in panel and network data models. In Chapter 1, I propose a semi-nonparametric panel data model with a latent group structure. I assume that individual parameters are heterogeneous across groups but homogeneous within a group while the group membership is unknown. I first approximate the infinite-dimensional function with a sieve expansion; then, I propose a Classifier-Lasso(C-Lasso) procedure to simultaneously identify the individuals’ membership and estimate the group-specific parameters. I show that: (i) the classification exhibits uniform consistency; (ii) C-Lasso and post-Lasso estimators achieve oracle properties so that they are asymptotically equivalent to infeasible estimators as if the group membership is known; and (iii) the estimators are consistent and asymptotically normally distributed. Simulations demonstrate an excellent finite sample performance of this approach in both classification and estimation.In Chapter 2 (joint with Wenyu Zhou), we study a nonparametric additive panel regression model with grouped heterogeneity. The model can be regarded as a natural extension to the heterogeneous panel model studied in Su, Shi, and Phillips (2016). We propose to estimate the nonparametric components using a sieve-approximation-based Classifier-Lasso method. We establish the asymptotic properties of the estimator and show that they enjoy the so-called oracle property. In addition, we present the decision rule for group classification and establish its consistency. Then, a BIC-type information criterion is developed to determine the group pattern of each nonparametric component. We further investigate the finite sample performance of the estimation method and the information criterion through Monte Carlo simulations. Results show that both work well. Finally, we apply the model and the estimation method to study the demand for cigarettes in the United States using panel data of

Various techniques have been applied to address residential self-selection (RSS) bias in estimating the impact of the built environment on travel behavior. We investigate how much the diverse results obtained are due to the method used, focusing on the most-often applied approaches of statistical control modeling, propensity score-based techniques, and sample selection modeling. First, a companion paper identifies 47 plausible ways to estimate a key quantity of interest, namely, the proportion of the total apparent effect of the built environment on travel behavior that is due to the built environment itself, which we call the “built environment proportion”, or BEP. Using a single sample, the present paper provides empirical estimates of 28 of the 47 BEP formulas, and creates and applies a framework for comparing those BEPs. We find that BEPs based on modular effects or treatment effects involving probability weights, and those applied to two-equation models, are among the ten best (based on the goodness of fit of the underlying models and on various measures of quality of the BEP). For our dataset, the ten best BEP averages fall between 0.506 and 0.772, with an overall average of 0.617. Thus, our best estimate is that for this application, about 38% of the total apparent effect of the built environment is due to RSS. However, the range of these averages indicates that even when controlling for as many confounding factors as possible, and even when using the best methods for computing the BEP, we still find substantial variation in the answer to the question: how much of the built environment's apparent total influence on travel behavior is due to self-selection?

This paper studies the evolution of the gender wage gap in Colombia at different points of wage distributions. Using DiNardo, Fortin, and Lemieux’s (1996) decomposition, we find that the enormous increase in female workers’ educational achievement has helped to reduce the gender gap, mainly at the top of the wage distribution. However, this effect has been countered by the reduction in the proportion of female workers in the public sector and those with indefinite contracts. Moreover, using the Arellano et al. (2017) methodology, we estimate the adjusted gender wage gap whilst controlling for sample selection. In both analysed years, our main finding was a clear glass-ceiling pattern that was slightly reduced in 2010., Este artigo estuda a brecha salarial por gênero na Colômbia em diferentes pontos da distribuição de salários. Com a decomposição de DiNardo et al. (1996), percebe-se que o enorme aumento nos objetivos educacionais das trabalhadoras femininas ajudou na redução da brecha de gênero, principalmente na alta da distribuição. Contudo, este efeito tem sido combatido pela redução na proporção de mulheres no setor público naquelas com contrato indefinido. Além do mais, com a metodologia de Arellano et al. (2016), estima-se uma brecha de gênero ajustada controlada por meio de viés de seleção. Percebe-se também um padrão de teto de cristal em ambos os anos analisados, com uma ligeira redução em 2010., Este artículo estudia la brecha salarial por género en Colombia en diferentes puntos de la distribución de salarios. Con la descomposición de DiNardo et al. (1996) se encuentra que el enorme incremento en el logro educacional de las trabajadoras femeninas ha ayudado a reducir la brecha de género, principalmente en la cola alta de la distribución. Sin embargo, este efecto ha sido contrarrestado por la reducción en la proporción de mujeres en el sector público y en aquellas con contrato indefinido. Además, con la metodología de Arellano et al. (2016) se estima la brecha de género ajustada controlando por sesgo de selección. Se encuentra un patrón de techo de cristal en ambos años analizados, que se ha reducido ligeramente en el 2010.

This article analyzes the determinants of attendance and frequency of assistance to cinema in Colombia. Using the data from the 2014 Cultural Consumption Survey, Heckman selection models are estimated to detect and correct selection biases that may arise in the research problem. The results indicate that although this activity is part of a model of mass cultural consumption, cultural capital and income, they remain important determinants in the consumption decisions of individuals. Factors such as age, marital status and occupation were also relevant. This work contributes to the applied literature in the country, to the extent that it allows a broader behavioral knowledge of the cinema consumer., En este artículo se analizan los determinantes de la asistencia y frecuencia de asistencia a cine en Colombia. Utilizando los datos de la Encuesta de Consumo Cultural 2014, se estiman modelos de selección de Heckman que permiten detectar y corregir los sesgos de selección que se pueden presentar en el problema de investigación. Los resultados indican que, aunque esta actividad se enmarca en un modelo de consumo cultural de masas, el capital cultural y los ingresos siguen siendo determinantes importantes en las decisiones de consumo de los individuos. Factores como la edad, el estado civil y la ocupación también resultaron relevantes. El presente trabajo contribuye a la literatura aplicada en el país en la medida en que permite un conocimiento más amplio del comportamiento del consumidor de cine.

This article analyzes the determinants of attendance and frequency of assistance to cinema in Colombia. Using the data from the 2014 Cultural Consumption Survey, Heckman selection models are estimated to detect and correct selection biases that may arise in the research problem. The results indicate that although this activity is part of a model of mass cultural consumption, cultural capital and income, they remain important determinants in the consumption decisions of individuals. Factors such as age, marital status and occupation were also relevant. This work contributes to the applied literature in the country, to the extent that it allows a broader behavioral knowledge of the cinema consumer., En este artículo se analizan los determinantes de la asistencia y frecuencia de asistencia a cine en Colombia. Utilizando los datos de la Encuesta de Consumo Cultural 2014, se estiman modelos de selección de Heckman que permiten detectar y corregir los sesgos de selección que se pueden presentar en el problema de investigación. Los resultados indican que, aunque esta actividad se enmarca en un modelo de consumo cultural de masas, el capital cultural y los ingresos siguen siendo determinantes importantes en las decisiones de consumo de los individuos. Factores como la edad, el estado civil y la ocupación también resultaron relevantes. El presente trabajo contribuye a la literatura aplicada en el país en la medida en que permite un conocimiento más amplio del comportamiento del consumidor de cine.

This paper studies the evolution of the gender wage gap in Colombia at different points of wage distributions. Using DiNardo, Fortin, and Lemieux's (1996) decomposition, we find that the enormous increase in female workers' educational achievement has helped to reduce the gender gap, mainly at the top of the wage distribution. However, this effect has been countered by the reduction in the proportion of female workers in the public sector and those with indefinite contracts. Moreover, using the Arellano et al. (2017) methodology, we estimate the adjusted gender wage gap whilst controlling for sample selection. In both analysed years, our main finding was a clear glass-ceiling pattern that was slightly reduced in 2010.

This paper compares three different word image representations as base for label free sample selection for word spotting in historical handwritten documents. These representations are a temporal pyramid representation based on pixel counts, a graph based representation, and a pyramidal histogram of characters (PHOC) representation predicted by a PHOCNet trained on synthetic data. We show that the PHOC representation can help to reduce the amount of required training samples by up to 69% depending on the dataset, if it is learned iteratively in an active learning like fashion. While this works for larger datasets containing about 1 700 images, for smaller datasets with 100 images, we find that the temporal pyramid and the graph representation perform better., open access

This paper compares three different word image representations as base for label free sample selection for word spotting in historical handwritten documents. These representations are a temporal pyramid representation based on pixel counts, a graph based representation, and a pyramidal histogram of characters (PHOC) representation predicted by a PHOCNet trained on synthetic data. We show that the PHOC representation can help to reduce the amount of required training samples by up to 69% depending on the dataset, if it is learned iteratively in an active learning like fashion. While this works for larger datasets containing about 1,700 images, for smaller datasets with 100 images, we find that the temporal pyramid and the graph representation perform better.

This paper compares three different word image representations as base for label free sample selection for word spotting in historical handwritten documents. These representations are a temporal pyramid representation based on pixel counts, a graph based representation, and a pyramidal histogram of characters (PHOC) representation predicted by a PHOCNet trained on synthetic data. We show that the PHOC representation can help to reduce the amount of required training samples by up to 69% depending on the dataset, if it is learned iteratively in an active learning like fashion. While this works for larger datasets containing about 1 700 images, for smaller datasets with 100 images, we find that the temporal pyramid and the graph representation perform better., open access

This paper compares three different word image representations as base for label free sample selection for word spotting in historical handwritten documents. These representations are a temporal pyramid representation based on pixel counts, a graph based representation, and a pyramidal histogram of characters (PHOC) representation predicted by a PHOCNet trained on synthetic data. We show that the PHOC representation can help to reduce the amount of required training samples by up to 69% depending on the dataset, if it is learned iteratively in an active learning like fashion. While this works for larger datasets containing about 1 700 images, for smaller datasets with 100 images, we find that the temporal pyramid and the graph representation perform better., open access

This paper compares three different word image representations as base for label free sample selection for word spotting in historical handwritten documents. These representations are a temporal pyramid representation based on pixel counts, a graph based representation, and a pyramidal histogram of characters (PHOC) representation predicted by a PHOCNet trained on synthetic data. We show that the PHOC representation can help to reduce the amount of required training samples by up to 69% depending on the dataset, if it is learned iteratively in an active learning like fashion. While this works for larger datasets containing about 1 700 images, for smaller datasets with 100 images, we find that the temporal pyramid and the graph representation perform better., open access

This paper compares three different word image representations as base for label free sample selection for word spotting in historical handwritten documents. These representations are a temporal pyramid representation based on pixel counts, a graph based representation, and a pyramidal histogram of characters (PHOC) representation predicted by a PHOCNet trained on synthetic data. We show that the PHOC representation can help to reduce the amount of required training samples by up to 69% depending on the dataset, if it is learned iteratively in an active learning like fashion. While this works for larger datasets containing about 1 700 images, for smaller datasets with 100 images, we find that the temporal pyramid and the graph representation perform better., open access

The current state-of-the-art, in terms of performance, for solving document image binarization is training artificial neural networks on pre-labelled ground truth data. As such, it faces the same issues as other, more conventional, classification problems; requiring a large amount of training data. However, unlike those conventional classification problems, document image binarization involves having to either manually craft or estimate the binarized ground truth data, which can be error-prone and time-consuming. This is where sample selection, the act of selecting training samples based on some method or metric, might help. By reducing the size of the training dataset in such a way that the binarization performance is not impacted, the required time spent creating the ground truth is also reduced. This thesis proposes a cluster-based sample selection method, based on previous work, that uses image similarity metrics and the relative neighbourhood graph to reduce the underlying redundancy of the dataset. The method is implemented with different clustering methods and similarity metrics for comparison, with the best implementation being based on affinity propagation and the structural similarity index. This implementation manages to reduce the training dataset by 46\% while maintaining a performance that is equal to that of the complete dataset. The performance of this method is shown to not be significantly different from randomly selecting the same number of samples. However, due to limitations in the random method, such as unpredictable performance and uncertainty in how many samples to select, the use of sample selection in document image binarization still shows great promise.

The current state-of-the-art, in terms of performance, for solving document image binarization is training artificial neural networks on pre-labelled ground truth data. As such, it faces the same issues as other, more conventional, classification problems; requiring a large amount of training data. However, unlike those conventional classification problems, document image binarization involves having to either manually craft or estimate the binarized ground truth data, which can be error-prone and time-consuming. This is where sample selection, the act of selecting training samples based on some method or metric, might help. By reducing the size of the training dataset in such a way that the binarization performance is not impacted, the required time spent creating the ground truth is also reduced. This paper proposes a cluster-based sample selection method that uses image similarity metrics and the relative neighbourhood graph to reduce the underlying redundancy of the dataset. The method, implemented with affinity propagation and the structural similarity index, reduces the training dataset on average by 49.57% while reducing the binarization performance only by 0.55%., open access

The current state-of-the-art, in terms of performance, for solving document image binarization is training artificial neural networks on pre-labelled ground truth data. As such, it faces the same issues as other, more conventional, classification problems; requiring a large amount of training data. However, unlike those conventional classification problems, document image binarization involves having to either manually craft or estimate the binarized ground truth data, which can be error-prone and time-consuming. This is where sample selection, the act of selecting training samples based on some method or metric, might help. By reducing the size of the training dataset in such a way that the binarization performance is not impacted, the required time spent creating the ground truth is also reduced. This thesis proposes a cluster-based sample selection method, based on previous work, that uses image similarity metrics and the relative neighbourhood graph to reduce the underlying redundancy of the dataset. The method is implemented with different clustering methods and similarity metrics for comparison, with the best implementation being based on affinity propagation and the structural similarity index. This implementation manages to reduce the training dataset by 46\% while maintaining a performance that is equal to that of the complete dataset. The performance of this method is shown to not be significantly different from randomly selecting the same number of samples. However, due to limitations in the random method, such as unpredictable performance and uncertainty in how many samples to select, the use of sample selection in document image binarization still shows great promise.

One line of critique from within the field of research in gifted education is that not enough studies are based on empirical evidence in examining the impact of educational interventions. However, further critique includes that within the empirical studies made there are limitations. The following article is divided in two parts, firstly, a review of 46 studies on acceleration or enrichment as part of research in gifted education is given, followed by critical reading of the same articles. The questions the article aim to answer are: What claims are made about the impact of acceleration and enrichment? What methodological limitations are exemplified in these studies? What recommendations can be made to researchers in gifted education in order to improve validity and reliability of research within the field? The majority of studies in the sample claim positive impact of the interventions in the form of acceleration or enrichment on a variety of factors such as academic abilities, self-concept and socio-emotional states among the students. However, the article claims a need for an increase in transparency of criteria for sample selection, as well as clarifications of definitions of significant concepts studied. Moreover, the article argues for a need for extended identifications procedures to avoid samples based on one instant tests only, or subjective teacher recommendations, as well as assumptions based on treating gifted students as homogenous. One suggestion to move forward is to also to use longitudinal studies basing the conclusions about impact to a greater extend on the development of these students as gifted learners.

The current state-of-the-art, in terms of performance, for solving document image binarization is training artificial neural networks on pre-labelled ground truth data. As such, it faces the same issues as other, more conventional, classification problems; requiring a large amount of training data. However, unlike those conventional classification problems, document image binarization involves having to either manually craft or estimate the binarized ground truth data, which can be error-prone and time-consuming. This is where sample selection, the act of selecting training samples based on some method or metric, might help. By reducing the size of the training dataset in such a way that the binarization performance is not impacted, the required time spent creating the ground truth is also reduced. This paper proposes a cluster-based sample selection method that uses image similarity metrics and the relative neighbourhood graph to reduce the underlying redundancy of the dataset. The method, implemented with affinity propagation and the structural similarity index, reduces the training dataset on average by 49.57% while reducing the binarization performance only by 0.55%., open access

The current state-of-the-art, in terms of performance, for solving document image binarization is training artificial neural networks on pre-labelled ground truth data. As such, it faces the same issues as other, more conventional, classification problems; requiring a large amount of training data. However, unlike those conventional classification problems, document image binarization involves having to either manually craft or estimate the binarized ground truth data, which can be error-prone and time-consuming. This is where sample selection, the act of selecting training samples based on some method or metric, might help. By reducing the size of the training dataset in such a way that the binarization performance is not impacted, the required time spent creating the ground truth is also reduced. This thesis proposes a cluster-based sample selection method, based on previous work, that uses image similarity metrics and the relative neighbourhood graph to reduce the underlying redundancy of the dataset. The method is implemented with different clustering methods and similarity metrics for comparison, with the best implementation being based on affinity propagation and the structural similarity index. This implementation manages to reduce the training dataset by 46\% while maintaining a performance that is equal to that of the complete dataset. The performance of this method is shown to not be significantly different from randomly selecting the same number of samples. However, due to limitations in the random method, such as unpredictable performance and uncertainty in how many samples to select, the use of sample selection in document image binarization still shows great promise.

The current state-of-the-art, in terms of performance, for solving document image binarization is training artificial neural networks on pre-labelled ground truth data. As such, it faces the same issues as other, more conventional, classification problems; requiring a large amount of training data. However, unlike those conventional classification problems, document image binarization involves having to either manually craft or estimate the binarized ground truth data, which can be error-prone and time-consuming. This is where sample selection, the act of selecting training samples based on some method or metric, might help. By reducing the size of the training dataset in such a way that the binarization performance is not impacted, the required time spent creating the ground truth is also reduced. This thesis proposes a cluster-based sample selection method, based on previous work, that uses image similarity metrics and the relative neighbourhood graph to reduce the underlying redundancy of the dataset. The method is implemented with different clustering methods and similarity metrics for comparison, with the best implementation being based on affinity propagation and the structural similarity index. This implementation manages to reduce the training dataset by 46\% while maintaining a performance that is equal to that of the complete dataset. The performance of this method is shown to not be significantly different from randomly selecting the same number of samples. However, due to limitations in the random method, such as unpredictable performance and uncertainty in how many samples to select, the use of sample selection in document image binarization still shows great promise.

The current state-of-the-art, in terms of performance, for solving document image binarization is training artificial neural networks on pre-labelled ground truth data. As such, it faces the same issues as other, more conventional, classification problems; requiring a large amount of training data. However, unlike those conventional classification problems, document image binarization involves having to either manually craft or estimate the binarized ground truth data, which can be error-prone and time-consuming. This is where sample selection, the act of selecting training samples based on some method or metric, might help. By reducing the size of the training dataset in such a way that the binarization performance is not impacted, the required time spent creating the ground truth is also reduced. This paper proposes a cluster-based sample selection method that uses image similarity metrics and the relative neighbourhood graph to reduce the underlying redundancy of the dataset. The method, implemented with affinity propagation and the structural similarity index, reduces the training dataset on average by 49.57% while reducing the binarization performance only by 0.55%., open access

The current state-of-the-art, in terms of performance, for solving document image binarization is training artificial neural networks on pre-labelled ground truth data. As such, it faces the same issues as other, more conventional, classification problems; requiring a large amount of training data. However, unlike those conventional classification problems, document image binarization involves having to either manually craft or estimate the binarized ground truth data, which can be error-prone and time-consuming. This is where sample selection, the act of selecting training samples based on some method or metric, might help. By reducing the size of the training dataset in such a way that the binarization performance is not impacted, the required time spent creating the ground truth is also reduced. This paper proposes a cluster-based sample selection method that uses image similarity metrics and the relative neighbourhood graph to reduce the underlying redundancy of the dataset. The method, implemented with affinity propagation and the structural similarity index, reduces the training dataset on average by 49.57% while reducing the binarization performance only by 0.55%., open access