Search

This study aims to implement the Shannon-fano Adaptive data compression algorithm on characters as input data. This study also investigates the data compression ratio, which is the ratio between the number of data bits before and after compression. The resulting program is tested by using black-box testing, measuring the number of character variants and the number of types of characters to the compression ratio, and testing the objective truth with the Mean Square Error (MSE) method. The description of the characteristics of the application made is done by processing data in the form of a collection of characters that have different types of characters, variants, and the number of characters. This research presents algorithm that support the steps of making adaptive Shannon-fano compression applications. The length of the character determines the variant value, compression ratio, and the number of input character types. Based on the results of test results, no error occurs according to the comparison of the original text input and the decompression results. A higher appearance frequency of a character causes a greater compression ratio of the resulting file; the analysis shows that a higher number of types of input characters causes a lower compression ratio, which proves that the proposed method in real-time data compression improves the effectiveness and efficiency of the compression process.

Compression as data coding technique has seen approximately 70 years of research and practical innovation. Nowadays, powerful compression tools with good trade-offs exist for a range of file formats from plain text to rich multimedia. Yet in the dilemma of cloud providers to reduce log data sizes as much as possible while having to keep as much as possible around for regulatory reasons and compliance processes, many companies are looking for smarter solutions beyond brute compression. In this paper, comprehensive applied research setting around network and system logs is introduced by comparing text compression ratios and performance. The benchmark encompasses 13 tools and 30 tool-configuration-search combinations. The tool and algorithm relationships as well as benchmark results are modelled in a graph. After discussing the results, the paper reasons about limitations of individual approaches and suitable combinations of compression with smart adaptive log file handling. The adaptivity is based on the exploitation of knowledge on format-specific compression characteristics expressed in the graph, for which a proof-of-concept advisor service is provided.

The rapid growth in the amount of data in the digital world leads to the need for data compression, and so forth, reducing the number of bits needed to represent a text file, an image, audio, or video content. Compressing data saves storage capacity and speeds up data transmission. In this paper, we focus on the text compression and provide a comparison of algorithms (in particular, entropy-based arithmetic and dictionary-based Lempel&ndash, Ziv&ndash, Welch (LZW) methods) for text compression in different languages (Croatian, Finnish, Hungarian, Czech, Italian, French, German, and English). The main goal is to answer a question: &rdquo, How does the language of a text affect the compression ratio?&rdquo, The results indicated that the compression ratio is affected by the size of the language alphabet, and size or type of the text. For example, The European Green Deal was compressed by 75.79%, 76.17%, 77.33%, 76.84%, 73.25%, 74.63%, 75.14%, and 74.51% using the LZW algorithm, and by 72.54%, 71.47%, 72.87%, 73.43%, 69.62%, 69.94%, 72.42% and 72% using the arithmetic algorithm for the English, German, French, Italian, Czech, Hungarian, Finnish, and Croatian versions, respectively.

В работе исследуется возможность применения метода автоматической классификации научных текстов на основе сжатия данных, успешно применявшегося к полным текстам научных статей к классификации текстов на основе аннотаций. Для классификации были использованы библиографические описания публикаций из базы данных Scopus. Сравнение результатов проводилось с использованием тематических рубрик Scopus. Выявлено, что построение обучающей выборки на основе высокоцитируемых публикаций улучшает качество классификации., The paper investigates the applicability of the method of automatic classification of scientific texts based on data compression, successfully applied to the full texts of scientific articles to classify texts based on annotations. For classification, bibliographic descriptions of publications from the Scopus database were used. A comparison of the results was carried out using subject areas from Scopus. It was revealed that the construction of a training set based on highly cited publications improves the quality of classification.

Kapacitet prostora za pohranu uređaja predstavlja jedan od ključnih faktora u razvoju informatike. S potrebom za većom količinom kapaciteta, dok se s jedne strane razvijao hardver sve većeg kapaciteta, s druge strane su nastale tehnike kompresije podataka te je počeo razvoj sve boljih i efikasnijih tehnika za razne tipova podataka. U ovom radu obrađene su osnove kompresije podataka, te objašnjeni neki od algoritama kompresije podataka na primjerima kompresije teksta. Pojedinačno su objašnjeni određeni formati ostalih nekoliko vrsta multimedijskih sadržaja te su objašnjeni koncepti ili prikazani postupci koje koriste pri kompresiji. Device storage capacity represents one of the key factors in the development of information technology. With the necessity for more capacity, while one side was developing hardware with more storage capacity, the other side invented data compression techniques and started developing better and more efficient techniques for various types of data. This paper deals with the basics od data compression and explains some of the data compression algorithms using text compression examples. Specific formats of the several other types of multimedia content are explained individually as well as the concepts or methods they use for compression.

В статье определяются роль и место вторичных информационных продуктов, созданных на основе иноязычных источников, в научно-исследовательской (и педагогической) работе магистрантов.

Bu çalışma, yazarı bilinmeyen bir dokümanının yazarını tahmin etmeyi amaçlamaktadır. Bunun için 6 farklı köşe yazarına ait 6 köşe yazısı öncelikle ön-işlem aşamasına sokulmuştur. Ardından bu metinler üzerinden n-gram (2-3) ile özellikler çıkarılmıştır. Çıkarılan özellikler üzerinden sistem 6 farklı makine öğrenmesi üzerinde çapraz geçerleme (10) ile test edilmiştir. Buraya kadar olan kısım literatürde şimdiye kadar uygulanmış olan yöntemdir. Bizim önerimiz ön işlem aşamasının ardından eldeki metinleri LZW algoritması ile kayıpsız sıkıştırarak özellik sayısını azaltmak ve bunun sistemin başarısı üzerindeki etkileri araştırmak üzerinedir. Ön-işlemden geçmiş olan metinler LZW algoritması ile binary (ikili) ve decimal (onlu) olarak sıkıştırılır. Sıkıştırmanın ardından n-gram (2-3) ile çıkarılan özellikler ile sistem 6 farklı makine öğrenmesi yönteminde test edilmiş ve çalışma sonuçları 5 farklı metrik için incelenmiştir. Yapılan çalışma sonucunda ikili olarak sıkıştırılmış metinler hem 2-gram hem de 3-gramda, 6 farklı makine öğrenmesi algoritmasında da daha iyi sonuçlar elde etmiştir. Random Tree ve Naïve bayes algoritmasında onlu sıkıştırma, ham verinin gerisinde kalsa da diğer 4 algoritmada daha iyi elde sonuçlar elde etmiş ama ortalama başarı değerlerinde geride kalmıştır. Yapılan çalışma sonucunda ikili sıkıştırma tüm metriklerinde diğer iki yönteme göre daha başarılıdır. Yapılan çalışmada yazar tanıma işlemi yapılmış olsa da önerilen bu yöntemin tüm metin sınıflandırma işlemlerinde kullanılabileceği düşünülmektedir. This study aims to estimate the author of an unknown document. For this purpose, first of all, six different columns of 6 different columnists were pre-processed. Then with n-grams (2-3) features were extracted from these texts. The system has been tested with 10-fold cross-validation on 6 different machine learning algorithms. This part of the study is the method that has been applied so far in the literature. Our suggestion is to reduce the number of features with the LZW algorithm and to investigate the effects on the success of the system. The pre-processed texts are compressed binary and decimal with the LZW algorithm. After compression, the system has been tested with 6 different machine learning algorithms, and the study results has been analyzed for 5 different metrics. As a result of the study, the compressed binary text has obtained better results in both 2-gram and 3-gram, for 6 different machine learning algorithms. In the Random-Tree and Naïve Bayes algorithm, decimal compression is behind the raw data. In the other four algorithms, it achieved better results but remained behind the average success values. As a result of the study, binary compression is more successful in all metrics than the other two methods. In the study, although the author recognition process has been done, it can be thought that the proposed method can be used in all text classification procedures.

Neural sequence-to-sequence models have been successfully applied to text compression. However, these models were trained on huge automatically induced parallel corpora, which are only available for a few domains and tasks. In this paper, we propose a novel interactive setup to neural text compression that enables transferring a model to new domains and compression tasks with minimal human supervision. This is achieved by employing active learning, which intelligently samples from a large pool of unlabeled data. Using this setup, we can successfully adapt a model trained on small data of 40k samples for a headline generation task to a general text compression dataset at an acceptable compression quality with just 500 sampled instances annotated by a human.

The widespread use of text databases and their exponential growth has boosted the interest in developing text compression techniques. These techniques aim at representing text collections using less space, but also at efficiently processing them by providing functionalities such as searching for words and phrases in the compressed text or decompressing just a portion of the text. In this paper, we study compression techniques from the perspective of the savings obtained in energy consumption when searching directly within the compressed text. Our results show that the use of text compression techniques can have an important influence on energy efficiency when the text is processed, for example, for word or phrase searches. Our evaluation compares the energy consumption of both compressed and uncompressed text searches, and the results show that compressing text databases can lead to important energy savings (around 50%).

Text compression is used to reduce the repetition of letters so that the storage space used becomes smaller and transmission becomes faster. Techniques in text compression that can be used are using lossless compression. Lossless compression is a type of compression where when the decompression process occurs no data is lost. The Even-Rodeh Code algorithm is a lossless type of compression algorithm that replaces the initial bit code of the data with a code from the Even-Rodeh Code algorithm so that it can produce smaller data sizes

ISBN ={978-3-95977-070-5}, ISSN ={1868-8969},; International audience; Given a set P of words, the Shortest Linear Superstring (SLS) problem is an optimization problem that asks for a superstring of $P$ of minimal length. SLS has applications in data compression, where a superstring is a compact representation of $P$, and in bioinformatics where it models the first step of genome assembly. Unfortunately SLS is hard to solve (NP-hard) and to closely approximate (MAX-SNP-hard). If numerous polynomial time approximation algorithms have been devised, few articles report on their practical performance. We lack knowledge about how closely an approximate superstring can be from an optimal one in practice. Here, we exhibit a linear time algorithm that reports an upper and a lower bound on the length of an optimal superstring. The upper bound is the length of a superstring. This algorithm can be used to evaluate beforehand whether one can get an approximate superstring whose length is close to the optimum for a given instance. Experimental results suggest that its approximation performance is orders of magnitude better than previously reported practical values. Moreover, the proposed algorithm remain efficient even on large instances and can serve to explore in practice the approximability of SLS.

We present the OpenCL GPU realization of the Burrows-Wheeler transform and describe the features which help achieve decent performance.

In this research paper we build a new, reliable, and sufficient algorithm for Arabic text language. The proposed algorithm should combine the features of the Huffman and Lempel Ziv algorithms, and is expected be able to reduce the general compression ratio . Our approach is different from Huffman algorithm in the sense that it assigns codes to n-gram symbols where n is a positive integer that is greater than or equal to one. Compared to Huffman algorithm, which assigns a code to each symbol individually, our approach is expected to assign codes to symbols in average. Our approach is different from Lempel Ziv algorithm in the sense that the size of dictionary that we build does not grow in an uncontrolled manner. The size of the dictionary is fixed and its size can be expected prior to process the text files that are to be compressed. This is because the size of each word in the dictionary we build is fixed and is equal to n. So for example, given that the number of different symbols in the text file at hand is m and that n is 2, the total number of entries in the dictionary that we propose to build will be m*m in the worst case.

The greedy approach to dictionary-based static text compression can be executed by a finite-state machine. When it is applied in parallel to different blocks of data independently, there is no lack of robustness even on standard large scale distributed systems with input files of arbitrary size. Beyond standard large scale, a negative effect on the compression effectiveness is caused by the very small size of the data blocks. A robust approach for extreme distributed systems is presented in this paper, where this problem is fixed by overlapping adjacent blocks and preprocessing the neighborhoods of the boundaries.

This paper presents CSGM2, a text preprocessing technique for compression purposes. It converts the original text into a word net (graph representation) and can retain the detailed contextual information such as word proximity. Specific directed graph is proposed to model this word net where words are stored in vertices and edges represent word transitions. The word net is fully capable of holding the natural word order in the original text and hence can be used directly for encoding purposes.

The concept of compression comes from the need to store data using as less space as possible and to ease transfer of data through a channel. The proposed algorithm deals with compression of text files using character replacement technique. For every string of length six, it is compressed by assigning a single character to it, maintaining a dictionary. The dictionary is used to decompress the encoded file. This gives a good compression ratio irrespective of the content of the text file.

Flexible parsing algorithm, a two-steps-greedy parsing algorithm for text factorisation, has been proved to be an optimal parsing for LZ78-like compressors in the case of constant-cost phrases [1] , [2] . Whilst in early implementations of LZ78-like compressors the phrases have constant cost, in common modern implementations the cost of the k-th phrase is ⌈ log 2 ⁡ k + C ⌉ where C is a real constant [3] , [4] . Indeed we show examples where Flexible parsing is not optimal under the above more realistic setting. In this paper we prove that, under the assumption that the cost of a phrase is block-wise constant and non-decreasing, the Flexible parsing is almost optimal. For almost optimal we mean that, for any text T, the difference between the sizes of the compressed text obtained by using a Flexible parsing and an optimal parsing is bounded by the maximal cost of a phrase in T, i.e. it is logarithmic in practical cases. Furthermore we investigate how an optimal parsing, and hence an almost optimal parsing, affects the rate of convergence to the entropy of LZ78-like compressors. We discuss some experimental results considering the ratio between the speed of convergence to the entropy of compressors with and without an optimal parsing. This ratio presents a kind of wave effect that increases as the entropy of a memoryless source decreases but it seems always to slowly converge to one. According to the theory, this wave can be a tsunami for some families of highly compressible strings and, although the optimal (and the almost optimal) parsing does not improve the asymptotical speed of convergence to the entropy, it can improve compression ratio, and hence the decoding speed, in many practical cases.

In this paper we analyze news text collections (clusters) via extracting their paraphrase headlines into a paraphrase graph and working with this graph. Our aim is to test whether news headline is an appropriate form of news text compression. Different types of news collections: dynamic, static and combined (both dynamic and static) clusters are analyzed and it is shown that their respective paraphrase graphs reflect the characteristics of the texts. We also automatically extract the most informationally important linked fragments of news texts, and these fragments characterize news texts as either informative, conveying some information, or publicistic ones, trying to affect the readers emotionally. It is shown that news headlines of the informative type do represent their respective compressed news reports.

Son yıllarda teknolojinin hızlı ilerleyişi ve yaygınlaşmasıyla teknolojinin getirdiği yenilikleri kullanmamak imkânsız hale gelmiştir. Bu yeniliklerden olan telefon, bilgisayar ve internet kullanımı yaygınlaştıkça insanların haberleşme ve bilgi güvenliği konusunda şüpheleri de artmıştır. Bu nedenle bilgi güvenliği çağımızın en önemli konularından biri haline gelmiştir.Klasik güvenlik yaklaşımlarında kullanılan şifreleme yani kriptografi algoritmaları bilginin gizliliğini sağlamaktadır. Fakat son zamanlarda giderek daha fazla üzerinde durulan `fark edilemezlik` ilkesini sağlayamamaktadır. Çünkü şifrelenmiş bir metinin üzerine yoğunlaşıldığı zaman gelişmiş bilgisayarlar sayesinde çözülebilmektedir. Dolayısıyla burada anahtar nokta `şüphe uyandırma` durumudur. Bu sebeple, gizli bir mesajın varlığının sezilememesini öncelikli hedefe koyan steganografi ve damgalama yöntemleri özellikle 2000'li yıllardan sonra ilgi odağı konumundadır. Bu tez çalışmasında uzamsal düzlemde en düşük değerlikli bit değiştirme yöntemine dayalı yeni bir görüntü steganografi yöntemi önerilmiştir. Önerilen yöntemi, steganografi'nin temel kriterlerinden biri olan güvenlik açısından geliştirmek için Rivest-Shamir-Adleman(RSA) şifreleme algoritması, gizlenebilecek mesaj kapasitesini artırmak için ise Lempel-Ziv-Welch, Arithmetic, Move-to-Front ve Deflate kayıpsız sıkıştırma algoritmaları önerilen yöntem ile birleştirilmiştir. Önerilen yöntemi ve mevcut en düşük değerlikli bit değiştirme yöntemini uygulamak için MATLAB GUI ortamında gizleme ve çıkarma için iki arayüz tasarlanmıştır. Sonuçlar incelendiğinde, önerilen yöntem ile steganografinin üç kriterinde de iyileşme gösterildiği görülmüştür. In recent years, with the rapid progress and spread of technology, it has become impossible not to use the innovations brought by the technology. As the use of telephone, computer and internet became more and more common, the doubts about people's communication and information security have increased. For this reason, information security has become one of the most important issues of our time. Cryptography algorithms used in classical security approaches provide the confidentiality of information. But in recent times it has been unable to achieve the principle of `unnoticeable` which is increasingly emphasized. Because when an encrypted text is focused on, it can be solved by advanced computers. So the key point here is `suspicion`. For this reason, steganography and watermarking methods, which put the invisibility of the existence of a confidential message as a primary goal, have become a focus of interest especially after 2000's.In this thesis, a new image steganography method based on the least significant bit substitution method in the spatial plane is proposed. The proposed method is combined with the Rivest-Shamir-Adleman(RSA) encryption algorithm to improve the security of the steganography, which is one of the requirements of steganography, and the Lempel-Ziv-Welch, Arithmetic, Move-to-Front and Deflate lossless compression algorithms to increase the message capacity that can be hidden. To implement the proposed method and the existing least significant bit substitution method, two interfaces for embedding and extracting processes has been designed with MATLAB GUI environment. When the results were examined, it was seen that the proposed method showed improvement in all three criteria of steganography. 84

Text compression is the task of identifying the main information in the source text to form a short single sentence. A broad approach is to find a path containing common vertices in the word graph model. The first issue of this approach is that the path finding algorithm can separate words from the phrase expressing a content. This leads to create new sentences having different meaning from the original ones. The second issue is that when an information is expressed by different words or phrases, called co-reference situations. Due to lacking of mechanism for handling this situation, the compression will be missing information. We propose in this paper a method to overcome the above issues. The core of new method is the improved graph model in which each vertex illustrates a phrase with its corresponding Part-of-Speech label. The intersection vertices of branches are results of mechanism for handling co-references. The compressing algorithm reduces the graph and forms the final sentence. We use ROUGE measure to compare with two word graph-based baselines. The experiment result shows that our method creates short sentences containing rich information.

Inverted indexes are mainstream in Information Retrieval systems and many compression techniques have been proposed. The purpose of this paper is to explore the compression efficiency on a two-level inverted index tailored for n-gram indices. We use two compression techniques Optimal PForDelta and IPC. Both techniques are applied to a previous work of us, that has focused on developing a threshold to efficiently store subsequences inside a one or two level inverted index, based on their number of occurrences inside a biological sequence. We study the performance of these two compression algorithms over different fluctuations of the threshold. The compression ratio of the OptPFD is affected by the changes in the threshold and is also efficient as in text documents. Whereas, IPC has a different performance for each threshold and it is more stable, although it is much less efficient than in text documents.

Study and evaluate the efficiency of LZW and BWT techniques for different categories of Arabic text files of different sizes. Compare these techniques on Arabic and English text files is introduced. Additional to exploiting morphological features of the Arabic language to improve performance of LZW techniques. We found that the enhanced LZW was the best one for all categories of the Arabic texts, then the LZW standard and BWT respectively.

Effective storage utilization is the key concept for better working of any operating system. Even operating systems used for mobile phones are not an exception for this fact. This paper proposes a technique for maximizing the utilization of the storage space present in mobile phones. Thus it is important to utilize the space occupied by SMS files in phone’s memory, which take maximum space. The objective involved is designing a semantic dictionary based on Intelligent Dictionary Based Encoding (IDBE) which provides a high text compression ratio to utilize the space in phone’s memory. When SMS file will be received, English words present in the text will be replaced by the respective short words in the designed semantic dictionary. Thus replacing English words by the respective short forms reduces the space occupied by the SMS file. The paper describes the IDBE Compression Techniques for SMS Text Compression.