Search

Latent Dirichlet allocation (LDA) is a popular method for analyzing large text corpora, but it suffers from instability due to its reliance on random initialization. This results in different outcomes for replicated runs, hindering reproducibility. To address this, we introduce LDAPrototype, a new approach for selecting the most representative LDA run from multiple replications on the same dataset. LDAPrototype enhances the reliability of LDA conclusions by ensuring greater similarity between replications compared to traditional LDA runs or models chosen based on perplexity or NPMI. A key feature of LDAPrototype is its use of a novel model similarity measure called S-CLOP (Similarity of multiple sets by Clustering with LOcal Pruning). It is based on topic similarities, for which we compare the usage of measures like the thresholded Jaccard coefficient, cosine similarity, Jensen-Shannon divergence, and rank-biased overlap. The effectiveness of LDAPrototype is demonstrated through its application to six real datasets, including newspaper articles and tweets. The results show improved reproducibility and reliability in topic modeling outcomes. LDAPrototype's approach is noteworthy for its practical applicability, comprehensibility, ease of implementation, and computational efficiency. Furthermore, the algorithm's concept can be generalized to other topic modeling procedures that characterize topics through word distributions, making it a versatile tool in text data analysis. [ABSTRACT FROM AUTHOR]

Latent Dirichlet allocation (LDA) is a popular method for analyzing large text corpora, but it suffers from instability due to its reliance on random initialization. This results in different outcomes for replicated runs, hindering reproducibility. To address this, we introduce LDAPrototype, a new approach for selecting the most representative LDA run from multiple replications on the same dataset. LDAPrototype enhances the reliability of LDA conclusions by ensuring greater similarity between replications compared to traditional LDA runs or models chosen based on perplexity or NPMI. A key feature of LDAPrototype is its use of a novel model similarity measure called S-CLOP (Similarity of multiple sets by Clustering with LOcal Pruning). It is based on topic similarities, for which we compare the usage of measures like the thresholded Jaccard coefficient, cosine similarity, Jensen-Shannon divergence, and rank-biased overlap. The effectiveness of LDAPrototype is demonstrated through its application to six real datasets, including newspaper articles and tweets. The results show improved reproducibility and reliability in topic modeling outcomes. LDAPrototype’s approach is noteworthy for its practical applicability, comprehensibility, ease of implementation, and computational efficiency. Furthermore, the algorithm’s concept can be generalized to other topic modeling procedures that characterize topics through word distributions, making it a versatile tool in text data analysis.

We consider two problems that are close in terms of formulation. The first one is that of clustering, i.e., partitioning the set of -dimensional Euclidean vectors into a given number of clusters with different types of centers so that the total variance would be minimum. Here, by variance we mean the sum of squared distances between the elements of the clusters and their centers. There are three types of centers: an arbitrary point (clearly, the centroid is the best choice), a point of the initial set (the so-called medoid), or a fixed point of the space given in advance.. The sizes of the clusters are also given as part of the input. The second problem under consideration is the problem of choosing a subset of vectors of fixed cardinality having the minimum sum of squared distances between its elements and the centroid. Polynomial-time approximation schemes (PTAS) are constructed for each of these problems. [ABSTRACT FROM AUTHOR]

• Key quality criteria for pixel-based composites identified and assessed. • New method for comprehensive and quantitative assessment of composites. • Over 16,000 Landsat imagery were mosaicked and analyzed over Europe. • Best Available Pixel (BAP) and Medoid composites were assessed. • BAP was more temporally consistent while Medoid was more radiometrically consistent. Remote sensing is one of the main sources of information for monitoring forest dynamics; however, surface reflectance is often not possible to accurately derive due to haze, cloud, or cloud shadow. Pixel-based composites are generated from multi-temporal images to cover the entire area of interest using several different methods. While the availability of free and open remote sensing data has further expanded the use of compositing approaches, to date a comprehensive methodology to assess the quality of these composites does not exist, nor is there a detailed set of compositing requirements to ensure consistent and reliable outputs to produce maps and statistics. Herein, we introduce a pixel-based composite assessment methodology based on five criteria: (i) number of valid observations and number of pixels with no available observations (data gaps), (ii) amount of unscreened clouds, cloud shadows, haze, or smoke (noise), (iii) radiometric consistency of the surface reflectance data, (iv) temporal proximity of pixels acquisition dates, and (v) spatial agreement of pixels acquisition dates. To test our methodology, we processed more than 16,000 Landsat images to generate and assess the Best Available Pixel (BAP) and the Medoid pixel-based composites for summer 2019 (2019-Jun-1 to 2019-Aug-31) over Europe, with a focus on the forested ecosystems. We found that BAP resulted in composites that were more temporally consistent, whereas the Medoid approach resulted in composites that were more radiometrically consistent. Our results illustrate that our assessment approach is effective for comprehensively assessing the quality of pixel-based composites and could be implemented when using composites to generate statistical estimates (i.e. forest area) and for assessing the performance of new compositing algorithms or for selecting an appropriate compositing approach for a specific application. [ABSTRACT FROM AUTHOR]

We consider the following 2-clustering problem. Given N points in Euclidean space, partition it into two subsets (clusters) so that the sum of squared distances between the elements of the clusters and their centers would be minimum. The center of the first cluster coincides with its centroid (mean) while the center of the second cluster should be chosen from the set of the initial points (medoid). It is known that this problem is NP-hard if the cardinalities of the clusters are given as a part of the input. In this paper we prove that the problem remains NP-hard in the case of arbitrary clusters sizes and suggest a 2-approximation polynomial-time algorithm for this problem.

Averaging GPS trajectories is needed in applications such as clustering and automatic extraction of road segments. Calculating mean for trajectories and other time series data is non-trivial and shown to be an NP-hard problem. medoid has therefore been widely used as a practical alternative and because of its (assumed) better noise tolerance. In this paper, we study the usefulness of the medoid to solve the averaging problem with ten different trajectory-similarity/-distance measures. Our results show that the accuracy of medoid depends mainly on the sample size. Compared to other averaging methods, the performance deteriorates especially when there are only few samples from which the medoid must be selected. Another weakness is that medoid inherits properties such as the sample frequency of the arbitrarily selected sample. The choice of the trajectory distance function becomes less significant. For practical applications, other averaging methods than medoid seem a better alternative for higher accuracy.

Averaging GPS trajectories is needed in applications such as clustering and automatic extraction of road segments. Calculating mean for trajectories and other time series data is non-trivial and shown to be an NP-hard problem. medoid has therefore been widely used as a practical alternative and because of its (assumed) better noise tolerance. In this paper, we study the usefulness of the medoid to solve the averaging problem with ten different trajectory-similarity/-distance measures. Our results show that the accuracy of medoid depends mainly on the sample size. Compared to other averaging methods, the performance deteriorates especially when there are only few samples from which the medoid must be selected. Another weakness is that medoid inherits properties such as the sample frequency of the arbitrarily selected sample. The choice of the trajectory distance function becomes less significant. For practical applications, other averaging methods than medoid seem a better alternative for higher accuracy. [ABSTRACT FROM AUTHOR]