Search

Annotated corpus can greatly assist in the natural language processing field. For example, computers can understand more of the document context, and indexing and clustering in information retrieval can be done precisely with less or no ambiguity of words. However, there are only a few annotated corpora in Malay language, which are not publicly shared. In this paper, we delve into analysing and annotating Malay translated hadith documents in terms of tagging and entities. There are three phases, which are manual filtering and cleaning, analysing the corpus and creating the benchmark. As the result, an analysis and benchmark of Malay translated hadith corpus were produced in term of part-of-speech and named entities tags that follows the Zipf’s law distribution.

Textual passwords are one of the most common methods of authentication and an important factor in systems security. Knowing the correct distribution of users’ passwords can play an important role in defining password policies and preventing various attacks. Culture and language can affect the pattern of users’ password selection and consequently, influence the vulnerability of passwords to guessing attacks. Therefore, knowing the distribution of English users’ passwords may not be appropriate for the security analysis of non-English users’ passwords. The main purpose of this paper is to analyze the passwords of Iranian users and investigating their differences from English-speaking users. The paper also examines the existence of Zipf’s law on Iranian passwords as the most well-known distribution for passwords. Password analysis of Iranian users shows that the popular password length between Iranian users and users of other countries is not much different, but in terms of the combination of characters used in the passwords, Iranian users are more inclined to use numeric passwords while English language users are more inclined to use passwords made up of alphabet. In this paper, Zipf’s law is reviewed on five datasets of Iranian users’ passwords using three different approaches including PDF, PDF with removing unpopular passwords and, CDF. Among these methods, in the CDF method, the passwords best matched with a Zipf’s law distribution between 0.02 and 0.07. Finally, the robustness of Iranians’ passwords to statistical guessing attacks has been measured and it is concluded that the passwords of Iranian users are more vulnerable to guessing attacks than English language users.

Zipf’s law defines an inverse proportion between a word’s ranking in a given corpus and its frequency in it, roughly dividing the vocabulary into frequent words and infrequent ones. Here, we stipulate that within a domain an author’s signature can be derived from, in loose terms, the author’s missing popular words and frequently used infrequent words. We devise a method, termed Latent Personal Analysis (LPA), for finding domain-based attributes for entities in a domain: their distance from the domain and their signature, which determines how they most differ from a domain. We identify the most suitable distance metric for the method among several and construct the distances and personal signatures for authors, the domain’s entities. The signature consists of both over-used terms (compared to the average) and missing popular terms. We validate the correctness and power of the signatures in identifying users and set existence conditions. We test LPA in several domains, both textual and non-textual. We then demonstrate the use of the method in explainable authorship attribution: we define algorithms that utilize LPA to identify two types of impersonation in social media: (1) authors with sockpuppets (multiple) accounts and (2) front-users accounts, operated by several authors. We validate the algorithms and employ them over a large-scale dataset obtained from a social media site with over 4000 users. We corroborate these results using temporal rate analysis. LPA can further be used to devise personal attributes in a wide range of scientific domains in which the constituents have a long-tail distribution of elements.

International audience; Since its inception, Shannon's information theory has attracted interest for the study of language and music. Recently, a wide range of converging studies have shown how efficient communication pervades language, from phonetics to syntax. Efficient principles imply that more resources should be assigned to highly informative items. For instance, average information content was shown to be a better predictor of word length than frequency, revisiting one of the famous Zipf's law. However , in spite of the success of the efficient communication framework in the study of language and speech, very little work has investigated its relevance in the analysis of music. Here, we examine the organization of harmonic information in two large corpora of Western music, one made of MIDI files directly sequenced from scores, and the other made of MIDI recordings of live performances of highly skilled piano players. We show that there is a clear positive relationship between (contextual) information content of harmonic sequences and two essential musical properties, namely duration and loudness: the more unexpected a harmonic event is, the longer and the louder it is.

The Zipf-Mandelbrot distribution serves as a mathematical model for ranked frequencies in many areas of scientific research, including linguistics. Many linguistic units, like e.g., words or word n...

The paper proposes an original methodology of authorship attribution based on the deviations from Zipf distribution and statistical data obtained with the help of a concordance program and computations performed in a table processor. The methodology involves finding distances between input texts and a reference text basing on deviations of stop-words frequencies. The results that have been achieved prove that the proposed methodology allows performing efficient authorship attribution and that it can be used in the educational process to develop student skills and competencies pertaining to natural language processing.

Efficient provision of Video-on-Demand (VoD) services requires that popular videos are stored in a cache close to users. Video popularity (defined by requested count) prediction is, therefore, important for optimal choice of videos to be cached. The popularity of a video depends on many factors and, as a result, changes dynamically with time. Accurate video popularity estimation that can promptly respond to the variations in video popularity then becomes crucial. In this paper, we analyze a method, called Minimal Inverted Pyramid Distance (MIPD), to estimate a video popularity measure called the Inverted Pyramid Distance (IPD). MIPD requires choice of a parameter, $k$ , representing the number of past requests from each video used to calculate its IPD. We derive, analytically, expressions to determine an optimal value for $k$ , given the requirement on ranking a certain number of videos with specified confidence. In order to assess the prediction efficiency of MIPD, we have compared it by simulations against four other prediction methods: Least Recency Used (LRU), Least Frequency Used (LFU), Least Recently/Frequently Used (LRFU), and Exponential Weighted Moving Average (EWMA). Lacking real data, we have, based on an extensive literature review of real-life VoD system, designed a model of VoD system to provide a realistic simulation of videos with different patterns of popularity variation, using the Zipf (heavy-tailed) distribution of popularity and a non-homogeneous Poisson process for requests. From a large number of simulations, we conclude that the performance of MIPD is, in general, superior to all of the other four methods.

Listeners incrementally process words as they hear them, progressively updating inferences about what word is intended as the phonetic signal unfolds in time. As a consequence, phonetic cues positioned early in the signal for a word are on average more informative about word-identity because they disambiguate the intended word from more lexical alternatives than cues late in the word. In this contribution, we review two new findings about structure in lexicons and phonological grammars, and argue that both arise through the same biases on phonetic reduction and enhancement resulting from incremental processing. (i) Languages optimize their lexicons over time with respect to the amount of signal allocated to words relative to their predictability: words that are on average less predictable in context tend to be longer, while those that are on average more predictable tend to be shorter. However, the fact that phonetic material earlier in the word plays a larger role in word identification suggests that languages should also optimize the distribution of that information across the word. In this contribution we review recent work on a range of different languages that supports this hypothesis: less frequent words are not only on average longer, but also contain more highly informative segments early in the word. (ii) All languages are characterized by phonological grammars of rules describing predictable modifications of pronunciation in context. Because speakers appear to pronounce informative phonetic cues more carefully than less informative cues, it has been predicted that languages should be less likely to evolve phonological rules that reduce lexical contrast at word beginnings. A recent investigation through a statistical analysis of a cross-linguistic dataset of phonological rules strongly supports this hypothesis. Taken together, we argue that these findings suggest that the incrementality of lexical processing has wide-ranging effects on the evolution of phonotactic patterns.

This paper applies fractal theory to research of green space in megacity parks due to the lack of a sufficient qualitative description of the scale structure of park green space, a quantifiable evaluation system, and operable planning methods in traditional studies. Taking Beijing, Shanghai, Guangzhou, and Shenzhen as examples, GIS spatial analysis technology and the Zipf model are used to calculate the fractal dimension (q), the goodness of fit (R2), and the degree of difference (C) to deeply interpret the connotation of indicators and conduct a comparative analysis between cities to reveal fractal characteristics and laws. The research results show that (1) the fractal dimension is related to the complexity of the park green space system, (2) the fractal dimension characterizes the hierarchical iteration of the park green space to a certain extent and reflects the internal order of the scale distribution, (3) the scale distribution of green space in megacity parks deviates from the ideal pyramid configuration, and (4) there are various factors affecting the scale structure of park green space, such as natural base conditions, urban spatial structure, and the continuation of historical genes working together. On this basis, a series of targeted optimization strategies are proposed.

In term of Chinese natural language processing, it exits one particular problem that how to choose the strategy of word segmentation, which commonly includes char-based and word-based. Targeted at sentiment analysis of short text comparing with long text, the word-based segmentation faces the other problem that there are the more ambiguous or unregistered words in context of short text. The feature extraction done by the different Chinese Word Segmentation impact the statistic distribution of features, and further the accuracy of sentiment analysis. This paper evaluates five Chinese segmentation strategy effect on Sentiment Analysis of Short Text. We chose two word-based Chinese Word Segmentation (CWS), and three char-based n-gram and made usage of Zipf’s law to quantify and present the result of word segmentation.

The mapping of lexical meanings to wordforms is a major feature of natural languages. While usage pressures might assign short words to frequent meanings (Zipf’s law of abbreviation), the need for a productive and open-ended vocabulary, local constraints on sequences of symbols, and various other factors all shape the lexicons of the world’s languages. Despite their importance in shaping lexical structure, the relative contributions of these factors have not been fully quantified. Taking a coding-theoretic view of the lexicon and making use of a novel generative statistical model, we define upper bounds for the compressibility of the lexicon under various constraints. Examining corpora from 7 typologically diverse languages, we use those upper bounds to quantify the lexicon’s optimality and to explore the relative costs of major constraints on natural codes. We find that (compositional) morphology and graphotactics can sufficiently account for most of the complexity of natural codes—as measured by code length., Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ISBN:978-1-954085-46-6

Starting from a view on language as a complex, hierarchically organized system composed of many parts that have many interactions, this paper investigates statistical relationships between the linguistic variables ‘phoneme inventory size’, ‘syllable complexity’, ‘length of words’, ‘length of clauses’, and the nonlinguistic variable ‘population size’. By analyzing parallel textual material of 61 languages (18 language families) we found strong positive correlations between phoneme inventory size and mean syllable complexity (measured as number of phonemes) and between phoneme inventory size and mean number of monosyllabic words. We observed significant negative correlations between phoneme inventory size and the mean length of words and the mean length of clauses, measured as number of syllables. We then correlated the linguistic complexity data with estimated speaker population sizes and could reveal that languages with more speakers tend to have * more phonemes per syllable, * shorter words in number of syllables, * a higher number of monosyllabic words, * and a higher number of words per clause. Moreover, we reproduce the results of former studies that found a positive correlation between population size and phoneme inventory size for our language sample. The findings are discussed in light of previous research and within the framework of Systemic Typology. We propose that syllable complexity is a key factor in the correlations identified in this study, and that Zipf’s law of Abbreviation explains the associations between ‘word length’, ‘phoneme inventory size’ and the extralinguistic variable ‘population size’.

Zipf's law establishes a scaling behavior for word frequencies in large text corpora. The appearance of Zipfian properties in vocabularies (viewed as an intermediate phase between referentially useless one-word systems and one-to-one word-meaning vocabularies) has been previously explained as an optimization problem for the interests of speakers and hearers. Remarkably, humanlike vocabularies can be viewed also as bipartite graphs. Thus, the aim here is double: within a bipartite-graph approach to human vocabularies, to propose a decentralized language game model for the formation of Zipfian properties. To do this, we define a language game in which a population of artificial agents is involved in idealized linguistic interactions. Numerical simulations show the appearance of a drastic transition from an initially disordered state towards three kinds of vocabularies. Our results open ways to study Zipfian properties in language, reconciling models seeing communication as a global minima of information entropic energies and models focused on self-organization.

Romanian is one of the understudied languages in computational linguistics, with few resources available for the development of natural language processing tools. In this paper, we introduce LaRoSeDa, a Large Romanian Sentiment Data Set, which is composed of 15,000 positive and negative reviews collected from one of the largest Romanian e-commerce platforms. We employ two sentiment classification methods as baselines for our new data set, one based on low-level features (character n-grams) and one based on high-level features (bag-of-word-embeddings generated by clustering word embeddings with k-means). As an additional contribution, we replace the k-means clustering algorithm with self-organizing maps (SOMs), obtaining better results because the generated clusters of word embeddings are closer to the Zipf's law distribution, which is known to govern natural language. We also demonstrate the generalization capacity of using SOMs for the clustering of word embeddings on another recently-introduced Romanian data set, for text categorization by topic., Comment: Accepted at EACL 2021

In the domain of Natural Language Generation and Processing, a lot of work is being done for text generation. As the machines become able to understand the text and language, it leads to a significant reduction in human involvement. Many sequence models show great work in generating human like text, but the amount of research work done to check the extent up to which their results match the man-made texts are limited in number. In this paper, the text is generated using Long Short Term Memory networks (LSTMs) and Generative Pretrained Transformer-2 (GPT-2). The text by neural language models based on LSTMs and GPT-2 follows Zipf’s law and Heap’s law, two statistical representations followed by every natural language generated text. One of the main findings is about the influence of parameter Temperature on the text produced. The LSTM generated text improves as the value of Temperature increases. The comparison between GPT-2 and LSTM generated text also shows that text generated using GPT-2 is more similar to natural text than that generated by LSTMs.

A central pursuit of micro- and nano-scale engineering is describing the small-scale structure of materials in terms of spatial relationships between positions of particles (representing, e.g., individual atoms). In order to carry out such analyses, molecular simulation practitioners have developed a large array of fast methods to compute spatial relationships on particle-resolved data sets. We have developed a computational workflow that directly applies molecular simulation methods to GIS building data, in which individual buildings (instead of, e.g., atoms) are treated as the particles of interest. In so doing, we enable efficient quantification of "urban textures" consisting of >> O(103) buildings. This interdisciplinary toolkit potentially opens the door to new vistas for urban-systems modeling. As one of a few early examples, we provide evidence for a novel scaling relationship between size and ordinal rank of building clusters within several American cities, reminiscent of Zipf's Law. This scaling relation suggests a new perspective on fractal-like organization of urban environments.

Empirical laws are the statistical laws that describe the relation between entities in a large dataset. They are readily found in nature, and findings have been proven by observations [1]. The primary objective of this study is to verify some of the empirical laws such as Zipf’s law, Mandelbrot’s approximation, and Heap’s law for Hindi language corpus. This involves collecting a corpus, performing text normalization, tokenizing it to get a list of words, identifying word types and their frequency, sorting and ranking the data based on frequency, and representing the relation between the frequency and rank of the word types to validate Zipf’s law and Mandelbrot’s approximation. For Heap’s law, the relation between the number of word types and tokens for different subsets of the corpus is considered. Based on our observations, the Hindi language satisfies the laws mentioned above.

Classification between machine-translated text and original text are often tokenized on vocabulary of the corpi. With N-grams larger than uni-gram, one can create a model that estimates a decision boundary based on word frequency probability distribution; however, this approach is exponentially expensive because of high dimensionality and sparsity. Instead, we let samples of the corpi be represented by part-of-speech tagging which is significantly less vocabulary. With less trigram permutations, we can create a model with its tri-gram frequency probability distribution. In this paper, we explore less conventional ways of approaching techniques for handling documents, dictionaries, and the likes.

This paper aims to research the applicability of Benford’s Law in Chinese texts. Firstly, the Chinese corpus was collected and word segmentation was performed. The distributions of the first digit of frequency were calculated for words, low-frequency words and single characters respectively in Chinese texts, and the relative entropy (Kullback-Leibler distance) between the distributions and the general Benford’s law. Secondly, the parameter value range of the Generalized Benford’s law was researched, and in view of the limitation of Zipf’s law that is only applicable to large amounts of data, we carried out a statistical analysis of small-scale data. Then, the experimental analysis of the probability of the first digit of the word frequency of the single character data was carried out to explore the applicability of the Generalized Benford’s law for single澡character data. Finally, the applicability of Benford’s law was investigated for artificially modified corpus. The results show that the words and characters in Chinese texts conform to the Benford’s law, and Benford’s law overcomes the limitation of Zipf’s law on the size of the data sets, and the Generalized Benford’s law has the ability to discriminate the natural quality of the corpus, which has important practical significance for Chinese information processing.

The use of background knowledge is largely unexploited in text classification tasks. This paper explores word taxonomies as means for constructing new semantic features, which may improve the performance and robustness of the learned classifiers. We propose tax2vec, a parallel algorithm for constructing taxonomy-based features, and demonstrate its use on six short text classification problems: prediction of gender, personality type, age, news topics, drug side effects and drug effectiveness. The constructed semantic features, in combination with fast linear classifiers, tested against strong baselines such as hierarchical attention neural networks, achieves comparable classification results on short text documents. The algorithm's performance is also tested in a few-shot learning setting, indicating that the inclusion of semantic features can improve the performance in data-scarce situations. The tax2vec capability to extract corpus-specific semantic keywords is also demonstrated. Finally, we investigate the semantic space of potential features, where we observe a similarity with the well known Zipf's law., Comment: Accepted at CSL journal

We present a new method to learn video representations from large-scale unlabeled video data. Ideally, this representation will be generic and transferable, directly usable for new tasks such as action recognition and zero or few-shot learning. We formulate unsupervised representation learning as a multi-modal, multi-task learning problem, where the representations are shared across different modalities via distillation. Further, we introduce the concept of loss function evolution by using an evolutionary search algorithm to automatically find optimal combination of loss functions capturing many (self-supervised) tasks and modalities. Thirdly, we propose an unsupervised representation evaluation metric using distribution matching to a large unlabeled dataset as a prior constraint, based on Zipf's law. This unsupervised constraint, which is not guided by any labeling, produces similar results to weakly-supervised, task-specific ones. The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods. Notably, it is also more effective than several label-based methods (e.g., ImageNet), with the exception of large, fully labeled video datasets., Comment: arXiv admin note: text overlap with arXiv:1906.03248

Knowledge extraction from text documents requires identifying and classifying semantic content. Utilizing an appropriate domain ontology can facilitate this process if words and phrases can be linked to the classes and relationships within the ontology. This paper presents an exemplar-based algorithm to link text to semantically similar classes within an ontology constructed for the chronic pain medicine domain. Human annotators linked classes to text segments within a random document set for construction of an exemplar dictionary, which we examined for completeness using Zipf plot analysis. An algorithm was created to use this dictionary on previously unseen text to form a map between sentence text and probable class assignments. We performed a 5 × 5 cross-validation between human and algorithm annotations and examined both ROC and precision versus recall curves to show that the algorithm can identify the many medical and biopsychosocial components from the texts. We briefly describe a use case for detecting pain relief from various interventions utilizing the word-by-class maps. We conclude that an exemplar-based method can be a valuable tool in knowledge extraction from texts that share similar construction, such as medical progress notes.

Evidence for compression, or minimization of code length, has been found across biological systems from genomes to human language and music. Two linguistic laws—Menzerath's Law (which states that longer sequences consist of shorter constituents) and Zipf's Law of abbreviation (a negative relationship between signal length and frequency of use)—are predictions of compression. It has been proposed that compression is a universal in animal communication, but there have been mixed results, particularly in reference to Zipf's Law of abbreviation. Like songbirds, male gibbons ( Hylobates muelleri ) engage in long solo bouts with unique combinations of notes which combine into phrases. We found strong support for Menzerath's Law as the longer a phrase, the shorter the notes. To identify phrase types, we used state-of-the-art affinity propagation clustering, and were able to predict phrase types using support vector machines with a mean accuracy of 74%. Based on unsupervised phrase type classification, we did not find support for Zipf's Law of abbreviation. Our results indicate that adherence to linguistic laws in male gibbon solos depends on the unit of analysis. We conclude that principles of compression are applicable outside of human language, but may act differently across levels of organization in biological systems.

The generally accepted that Zipf-Distribution is a convinced access pattern for text-based Web. However, with the dramatic increasement of VoD media traffic on the Internet such as Flash P2P, the inconsistency between the access patterns of media objects and the Zipf model has been researched by many scholars. In this paper, we have studied a large variety of media work-loads collected from both browser and server sides in Adobe Flash P2P systems which applied in Youku, Youtube, etc. Through extensive analysis and modeling. And found the object reference ranks of all these workloads follow the logistic (LOG) distribution despite their different media systems and delivery methods by extensive analysis and modeling. This mean it does not follow long tail effect; Furthermore, we have constructed mathematical models which can applied in access pattern in FlashP2P traffic. By analyzing the model of media traffic access, it is possible to better describe the user’s access mode. Meantime, it is very suitable for the configuration and allocation of network resources which can be used more efficiently.

Purpose This paper aims to address the challenge of analysing the features of negative sentiment tweets. The method adopted in this paper elucidates the classification of social network documents and paves the way for sentiment analysis of tweets in further research. Design/methodology/approach This study classifies negative tweets and analyses their features. Findings Through negative tweet content analysis, tweets are divided into ten topics. Many related words and negative words were found. Some indicators of negative word use could reflect the degree to which users release negative emotions: part of speech, the density and frequency of negative words and negative word distribution. Furthermore, the distribution of negative words obeys Zipf’s law. Research limitations/implications This study manually analysed only a small sample of negative tweets. Practical implications The research explored how many categories of negative sentiment tweets there are on Twitter. Related words are helpful to construct an ontology of tweets, which helps people with information retrieval in a fixed research area. The analysis of extracted negative words determined the features of negative tweets, which is useful to detect the polarity of tweets by machine learning method. Originality/value The research provides an initial exploration of a negative document classification method and classifies the negative tweets into ten topics. By analysing the features of negative tweets, related words, negative words, the density of negative words, etc. are presented. This work is the first step to extend Plutchik’s emotion wheel theory into social media data analysis by constructing filed specific thesauri, referred to as local sentimental thesauri.

In this article, we summarize the methodology and the results of our 2-year-long efforts to construct a comprehensive WordNet for Turkish. In our approach, we mine a dictionary for synonym candidate pairs and manually mark the senses in which the candidates are synonymous. We marked every pair twice by different human annotators. We derive the synsets by finding the connected components of the graph whose edges are synonym senses. We also mined Turkish Wikipedia for hypernym relations among the senses. We analyzed the resulting WordNet to highlight the difficulties brought about by the dictionary construction methods of lexicographers. After splitting the unusually large synsets, we used random walk-based clustering that resulted in a Zipfian distribution of synset sizes. We compared our results to BalkaNet and automatic thesaurus construction methods using variation of information metric. Our Turkish WordNet is available online. Publisher's Version

Using the Google Ngram Corpora for six different languages (including two varieties of English), a large-scale time series analysis is conducted. It is demonstrated that diachronic changes of the parameters of the Zipf–Mandelbrot law (and the parameter of the Zipf law, all estimated by maximum likelihood) can be used to quantify and visualize important aspects of linguistic change (as represented in the Google Ngram Corpora). The analysis also reveals that there are important cross-linguistic differences. It is argued that the Zipf–Mandelbrot parameters can be used as a first indicator of diachronic linguistic change, but more thorough analyses should make use of the full spectrum of different lexical, syntactical and stylometric measures to fully understand the factors that actually drive those changes.

We investigate the predictive capability of mathematical models of the type-token relationship applied to the vocabulary growth profiles of selected English language documents. We compare the existing Good-Toulmin and Heaps formulae with an alternative approach based on Bernoulli trial word selection from a fixed finite vocabulary using the Zipf and Zipf-Mandelbrot probability distributions. We make two major observations: firstly, while the Zipf-Mandelbrot model makes better predictions of vocabulary growth than the Zipf model, the optimized parameters of the latter correlate better than those of the former with statistics gleaned independently from the data. Secondly, the mean of the Zipf-Mandelbrot, Good-Toulmin and Heaps models provides a more consistent and unbiased prediction of vocabulary than any individual model alone.

Despite three decades of intensive research efforts, it remains an open question as to what is the underlying distribution of user-generated passwords. In this paper, we make a substantial step forward toward understanding this foundational question. By introducing a number of computational statistical techniques and based on 14 large-scale data sets, which consist of 113.3 million real-world passwords, we, for the first time, propose two Zipf-like models (i.e., PDF-Zipf and CDF-Zipf) to characterize the distribution of passwords. More specifically, our PDF-Zipf model can well fit the popular passwords and obtain a coefficient of determination larger than 0.97; our CDF-Zipf model can well fit the entire password data set, with the maximum cumulative distribution function (CDF) deviation between the empirical distribution and the fitted theoretical model being 0.49%~4.59% (on an average 1.85%). With the concrete knowledge of password distributions, we suggest a new metric for measuring the strength of password data sets. Extensive experimental results show the effectiveness and general applicability of the proposed Zipf-like models and security metric.

The present paper employs a synergetic approach to exploring the relationship between the length and frequency among English multiword formulaic sequences. It attempts to test whether Zipf’s assumption on the inverse relationship between word length and word frequency, as well as the synergetic model constructed at lexical level, can be extended to multiword formulaic sequences. A corpus-driven approach is adopted to acquire sufficient data. Results show partial applicability of the assumption to a small part of formulaic sequences: sequences of words with whole structure and complete meaning. However, the majority of formulaic sequences, lexical bundles, are proved to be an exception to the conventional rule. The paper tries to offer explanation by exploring further the features of lexical bundles. It concludes that the synergetic system of length and frequency among lexical bundles is operated under the compromise between the requirements of minimization of production efforts and the requirement...

Humans can extract co-occurrence regularities from their environment, and use them for learning. This statistical learning ability (SL) has been studied extensively as a way to explain how we learn the structure of our environment. These investigations have illustrated the impact of various distributional properties on learning. However, almost all SL studies present the regularities to be learned in uniform frequency distributions where each unit (e.g., image triplet) appears the same number of times: While the regularities themselves are informative, the appearance of the units cannot be predicted. In contrast, real-world learning environments, including the words children hear and the objects they see, are not uniform. Recent research shows that word segmentation is facilitated in a skewed (Zipfian) distribution. Here, we examine the domain-generality of the effect and ask if visual SL is also facilitated in a Zipfian distribution. We use an existing database to show that object combinations have a skewed distribution in children's environment. We then show that children and adults showed better learning in a Zipfian distribution compared to a uniform one, overall, and for low-frequency triplets. These results illustrate the facilitative impact of skewed distributions on learning across modality and age; suggest that the use of uniform distributions may underestimate performance; and point to the possible learnability advantage of such distributions in the real-world.

We investigate the impact of the content popularity of video streams in a local region and its impact on the CDN deployment cost. We have gathered real-world data from four popular classroom video streaming sites and analyzed the content caching to optimize the CDN deployment. Our traces contain metadata of around 31 thousand educational videos and approximately 100 million views in the education category of YouTube. From this analysis, we assert that region based content (e.g., NPTEL, India) follows Zipf law with low popularity exponent, and it is the region-based content popularity which most significantly impacts the CDN deployment cost.

With the development and popularity of the Internet, Internet traffic has increased dramatically. Numerous studies have shown that video accounts for a large percentage of Internet traffic, and this percentage is still rising in the future. A good understanding of user behavior in online video systems can help us design, configure and manage video content distribution to alleviate network stress. In this paper, we did a detailed analysis of user behavior data for Internet video. Our research shows that the user's daily access and online pattern of users have a fixed pattern, and the user's access behavior conforms to Zipf's law. Besides, we optimized the fit of Zipf-like distribution of video's popularity. Finally, we built a reliable simulation system that simulates user behavior data. Overall, we believe that the results presented in this paper are very important and valuable to the whole network.

This second chapter exposes some useful background information that will be useful over the entire book. In particular, this chapter presents and defines precisely the notions of word-type, word-token, and lemma. The mathematical notation used in the entire book is also presented and commented. A running example (the Federalist Papers) is described and will serve to illustrate the concepts and computations done in the book. Next, the most important overall stylometric measurements are explained and numerical examples are provided. In this case, the Zipf’s Law and various vocabulary richness measures (e.g., type-token ratio (TTR), Herdan’s C, Yule’s K) are discussed. Finally, other global stylistic measurements are presented, such as lexical density (LD), percentage of big words (BW), or the mean sentence length (MSL).

TF-IDF is one of the most commonly used weighting metrics for measuring the relationship of words to documents. It is widely used for word feature extraction. In many research and applications, the thresholds of TF-IDF for selecting relevant words are only based on trial or experiences. Some cut-off strategies have been proposed in which the thresholds are selected based on Zipf’s law or feedbacks from model performances. However, the existing approaches are restricted in specific domains or tasks, and they ignore the imbalance of the number of representative words in different categories of documents. To address these issues, we apply game-theoretic shadowed set model to select the word features given TF-IDF information. Game-theoretic shadowed sets determine the thresholds of TF-IDF using game theory and repetition learning mechanism. Experimental results on real world news category dataset show that our model not only outperforms all baseline cut-off approaches, but also speeds up the classification algorithms.

Availability of the text in different languages has become possible, as almost all websites have offered multilingual option. Hindi is considered as official language in one of the states of India. Hindi text analysis is dominated by the corpus of stories and poems. Before performing any text analysis token extraction is an important step and supports many applications like text summarization , categorizing text and so on. Token extraction is a part of Natural language processing (NLP). NLP includes many steps such as preprocessing the corpus, lemmatization and so on. In this paper the tokens are extracted by two methods and on two corpora. BaSa, a context-based term extraction technique having different NLP activities, e.g. Term Frequency Inverse Document Frequency (TF-IDF) and Zipf ‘s law are used to count and compare extracted tokens. Further token comparison between both of the methods is achieved. The corpus contains proses and verses of Hindi as well as the Marathi language. Common tokens from corpora of verses and proses of Marathi as well as Hindi are identified to prove that both of them behave same as per as NLP activities are concerened. The betterment of BaSa over Zipf’s law is proved. Hindi Corpus includes 820 stories and 710 poems and Marathi corpus includes 610 stories and 505 poems.

We develop an economic model of an offline password cracker which allows us to make quantitative predictions about the fraction of accounts that a rational password attacker would crack in the event of an authentication server breach. We apply our economic model to analyze recent massive password breaches at Yahoo!, Dropbox, LastPass and AshleyMadison. All four organizations were using key-stretching to protect user passwords. In fact, LastPass' use of PBKDF2-SHA256 with $10^5$ hash iterations exceeds 2017 NIST minimum recommendation by an order of magnitude. Nevertheless, our analysis paints a bleak picture: the adopted key-stretching levels provide insufficient protection for user passwords. In particular, we present strong evidence that most user passwords follow a Zipf's law distribution, and characterize the behavior of a rational attacker when user passwords are selected from a Zipf's law distribution. We show that there is a finite threshold which depends on the Zipf's law parameters that characterizes the behavior of a rational attacker -- if the value of a cracked password (normalized by the cost of computing the password hash function) exceeds this threshold then the adversary's optimal strategy is always to continue attacking until each user password has been cracked. In all cases (Yahoo!, Dropbox, LastPass and AshleyMadison) we find that the value of a cracked password almost certainly exceeds this threshold meaning that a rational attacker would crack all passwords that are selected from the Zipf's law distribution (i.e., most user passwords). This prediction holds even if we incorporate an aggressive model of diminishing returns for the attacker (e.g., the total value of $500$ million cracked passwords is less than $100$ times the total value of $5$ million passwords). See paper for full abstract., Comment: IEEE Symposium on Security and Privacy (S&P) 2018

Two prominent statistical laws in language and other complex systems are Zipf’s law and Heaps’ law. We investigate the extent to which these two laws apply to the linguistic domain of phonotactics—that is, to sequences of sounds. We analyze phonotactic sequences with different lengths within words and across word boundaries taken from a corpus of spoken English (Buckeye). We demonstrate that the expected relationship between the two scaling laws can only be attested when boundary spanning phonotactic sequences are also taken into account. Furthermore, it is shown that Zipf’s law exhibits both high goodness-of-fit and a high scaling coefficient if sequences of more than two sounds are considered. Our results support the notion that phonotactic cognition employs information about boundary spanning phonotactic sequences.

This paper presents innovative method to extract stop words from large Urdu text. Stop words are less meaningful words in natural language that slow down language processing and affect language analysis negatively. For language analysis, stop words are removed first to ensure fast data processing. But for Urdu language, there is no reliable stop words removal method. In this work, we applied Zipf's law of two factors dependency with least effort approach to spot stop words in Urdu language corpus. Urdu corpus is specifically created for this research. All Urdu text processing and investigation is carried out in Python 3. 4. Previous work for stop words removal is also investigated and proved less helpful. By using Zipfian approach, out of 500 high frequency words, 358 words are identified as stop words. It is observed that by only focusing on 0.01% of large corpus, almost all the stop words can be spotted to create a stop words list with least manual effort. Furthermore, statistical patterns in stop words, content words, stop words vs content words ratio in data samples and dependency of stop words and content words over data size is also examined. In terms of data size, frequency and ranks, Zipf's law and Heap's law coexist in Urdu stop words. Stop words tend to follow some predictable and measurable patterns that can lead to reliable probabilistic methods for Urdu processing. This deterministic approach provides a strong research ground to explore stop words in Urdu text statistically.

Analytic queries enable sophisticated large-scale data analysis within many commercial, scientific and medical domains today. Data skew is a ubiquitous feature of these real-world domains. In a retail database, some products are typically much more popular than others. In a text database, word frequencies follow a Zipf distribution with a small number of very common words, and a long tail of infrequent words. In a geographic database, some regions have much higher populations (and therefore data measurements) than others. Current systems do not make the most of caches for exploiting skew. In particular, a whole cache line may remain cache resident even though only a small part of the cache line corresponds to a popular data item. In this paper, we propose a novel index structure for repositioning data items to concentrate popular items into the same cache lines. The net result is better spatial locality, and better utilization of limited cache resources. We develop a theoretical model for analyzing the cache utilization, and implement database operators that are efficient in the presence of skew. Our experimental evaluation on real and synthetic data shows that exploiting skew can significantly improve in-memory query performance. In some cases, our techniques can speed up queries by over an order of magnitude.

Douyin, internationally known as TikTok, has become one of the most successful short-video platforms. To maintain its popularity, Douyin has to provide better Quality of Experience (QoE) to its growing user base. Understanding the characteristics of Douyin videos is thus critical to its service improvement and system design. In this paper, we present an initial study on the fundamental characteristics of Douyin videos based on a dataset of over 260 thousand short videos collected across three months. The characteristics of Douyin videos are found to be significantly different from traditional online videos, ranging from video bitrate, size, to popularity. In particular, the distributions of the bitrate and size of videos follow Weibull distribution. We further observe that the most popular Douyin videos follow Zifp's law on video popularity, but the rest of the videos do not. We also investigate the correlation between popularity metrics used for Douyin videos. It is found that the correlation between the number of views and the number of likes are strong, while other correlations are relatively low. Finally, by using a case study, we demonstrate that the above findings can provide important guidance on designing an efficient edge caching system.

Being one of the most common empirical regularities, the Zipf’s law for word frequencies is a power law relation between word frequencies and frequency ranks of words. We quantitatively study semantic uncertainty of words through non-point distribution-based word embeddings and reveal the Zipfian regularities. Uncertainty of a word can increase due to polysemy, the word having “broad” meaning (such as the relation between broader emotion and narrower exasperation) or a combination of both. Variances of Gaussian embeddings are utilized to quantify the extent a word can be used in different senses or contexts. By using the variance information embedded in the non-point Gaussian embeddings, we quantitatively show that semantic breadth of words also exhibits Zipfian patterns, when polysemy is controlled. This outcome is complementary to Zipf’s law of meaning distribution and the related meaning-frequency law by indicating the existence of Zipfian patterns: more frequent words tend to be generic while less frequent ones tend to be specific. Results for two languages, English and Turkish that belong to different language families, are also provided. Such regularities provide valuable information to extract and understand relationships between semantic properties of words and word frequencies. In various applications, performance improvements can be obtained by employing these regularities. We also propose a method that leverages the Zipfian regularity to improve the performance of baseline textual entailment detection algorithms. To the best of our knowledge, our approach is the first quantitative study that uses Gaussian embeddings to examine the relationships between word frequencies and semantic breadth.

Zipf’s law describes the relationship between the frequencies of words in a corpus and their rank. Its most basic form is a simple series, indicating that the frequency of a word is inversely proportional to its rank: 1/2 , 1/3 , 1/ 4 ,... The past two decades have seen the emergence of usage-based and cognitive approaches to language study. A key observation of these approaches, along with the importance of frequency, is that speech differs in substantial and structural ways from writing. Yet, except for a few older analyses performed on very small corpora, most studies of Zipf’s law have been done on written corpora. Further, a judgement of Zifianness in much of this work is based on loose and informal criteria. In fact, sophisticated statistical techniques have been developed for curve fitting in recent years in the mathematics and physics literature . These include the use of the Kolmogorov-Smirnov statistic, along with maximum likelihood estimation to generate p-values and the use of the complementary error function for normal distributions. The latter helps determine if a corpus, failing a Zipfian fit, might be better described by another distribution. In this paper, we will: Show that three corpora of recorded speech follow a power law distribution using rigorous statis- tical techniques: Buckeye, Santa Barbara, MiCase Describe preliminary results showing that the techniques outlined in this paper may be useful in the diagnoses of those conditions that can include disordered speech. Explain how to do the analyses described in this paper. Explain how to download and use the R/Python code we have written and packaged as the Zipf Tool Kit

The paper examines Lithuanian texts of different authors and genres. The main points ofinterest – the number of words, the number of different words and word frequencies. Structural type distributionand Zipf’s law are applied for describing the frequency distribution of words in the text. It is obvious that thelexical diversity of any text can be defined by different words that are used in the text, also called vocabulary.It is shown that the information contained in a reduced vocabulary is enough for dividing the texts analyzedin this article into groups by genre and author using a hierarchical clustering method. In this case, distancesbetween clusters are measured using the Jaccard distance measure, and clusters are aggregated using the Wardmethod.

ContextApplication Programming Interfaces (APIs) facilitate the use of programming languages. They define sets of rules and specifications for software programs to interact with. The design of language API is usually artistic, driven by aesthetic concerns and the intuitions of language architects. Despite recent studies on limited scope of API usage, there is a lack of comprehensive, quantitative analyses that explore and seek to understand how real-world source code uses language APIs. ObjectiveThis study aims to understand how APIs are employed in practical development and explore their potential applications based on the results of API usage analysis. MethodWe conduct a large-scale, comprehensive, empirical analysis of the actual usage of APIs on Java, a modern, mature, and widely-used programming language. Our corpus contains over 5000 open-source Java projects, totaling 150 million source lines of code (SLoC). We study the usage of both core (official) API library and third-party (unofficial) API libraries. We resolve project dependencies automatically, generate accurate resolved abstract syntax trees (ASTs), capture used API entities from over 1.5 million ASTs, and measure the usage based on our defined metrics: frequency, popularity and coverage. ResultsOur study provides detailed quantitative information and yield insight, particularly, (1) confirms the conventional wisdom that the usage of APIs obeys Zipf distribution; (2) demonstrates that core API is not fully used (many classes, methods and fields have never been used); (3) discovers that deprecated API entities (in which some were deprecated long ago) are still widely used; (4) evaluates that the use of current compact profiles is under-utilized; (5) identifies API library coldspots and hotspots. ConclusionsOur findings are suggestive of potential applications across language API design, optimization and restriction, API education, library recommendation and compact profile construction.

This paper proposes a novel methodology for delineating urban areas based on a machine learning algorithm that groups buildings within portions of space of sufficient density. To do so, we use the precise geolocation of all 12 million buildings in Spain. We exploit building heights to create a new dimension for urban areas, namely, the vertical land, which provides a more accurate measure of their size. To better understand their internal structure and to illustrate an additional use for our algorithm, we also identify employment centers within the delineated urban areas. We test the robustness of our method and compare our urban areas to other delineations obtained using administrative borders and commuting-based patterns. We show that: 1) our urban areas are more similar to the commuting-based delineations than the administrative boundaries but that they are more precisely measured; 2) when analyzing the urban areas' size distribution, Zipf's law appears to hold for their population, surface and vertical land; and 3) the impact of transportation improvements on the size of the urban areas is not underestimated.

The possibilities of the semantic analysis technique developed in linguistics and used in working with texts for the tasks of psychological study of the professional activities of specialists are substantiated. On the example of the study of risk perceptions in the work of police psychologists and practicing psychologists and teachers of the university, the advantages and limitations of the methodology are shown. The similarity of the structures of perceptions of the risk of professional activity among psychologists has been ascertained: emotional burnout and professional deformation, the use of psychological assistance methods, fear of error, legal liability, proper interaction with the client, relationships with colleagues. Double differences in the levels of professional risk reflection, the banality of risk ideas, and the harmony of the structure of representations (“Zipf’s Law”) are revealed. All indicators are optimal for police psychologists. It is concluded that it is advisable to use the methods of semantic analysis as an express risk diagnosis in professional activities.

With the increase in the number of character assassination and fake news recently happening in Nigeria, we combine Zipf’s law and Benford’s law to analyse and detect fake news. The problem of fake news has become one of the most prominent issues in Nigeria recently. In this chapter, the challenges fake news poses to Nigeria is briefly presented. Due to these challenges, we propose the combination of Benford’s law and Zipf’s law in news analysis such that the hybrid of the two laws will obey the Power law for real news and deviate for fake news. We carried out various tests on different real news sources and the result shows that real news obeys the Power law. We, therefore, propose that fake news should not obey the Power law even though we could not test on fake news sources because of the lack of verified fake news dataset.

With rapid increase in the use of traffic-intensive applications, approaches that improve users experience by reducing delay are recently receiving enormous attention. Caching in Device-to-Device (D2D) networks, in particular, is considered as an effective technique to improve the service quality of the network. In this paper, an agglomerative hierarchical clustering algorithm is proposed for a cache-assisted D2D communication network. The algorithm considers users preferences and groups them into the same cluster based on the similarity of their requested content. An optimal caching strategy has been applied and the cache hit probability has further being optimized within each cluster. Performance of the algorithm has been examined in different clusters, considering both sparse and dense user environments. Simulation results show that the cache hit probability within each cluster is higher for higher Zipf parameter, denser domains, and larger number of participating devices. The D2D cache hit probability has also been examined with changing number of clusters under a base station. In this scenario, the results show that the clustering based D2D cache hit probability is higher than the non-clustered case, and the cache hit probability increases with increasing number of clusters.

We study the problem of estimating the total volume of queries of a specific domain, which were submitted to the Google search engine in a given time period. Our statistical model assumes a Zipf’s law distribution of the population in the reference domain, and a non- uniform or noisy sampling of queries. Parameters of the distribution are estimated using nonlinear least square regression. Estimations with errors are then derived for the total number of queries and for the total number of searches (volume). We apply the method on the recipes and cooking domain, where a sample of queries is collected by crawling popular Italian websites specialized on this domain. The relative volumes of queries in the sample are computed using Google Trends, and transformed to absolute frequencies after estimating a scaling factor. Our model estimates that the volume of Italian recipes and cooking queries submitted to Google in 2017 and with at least 10 monthly searches consists of 7.2B searches.