Search

Graphical abstractThis work proposes a scheme to improve the self-organizing map algorithm. This is the overall flowchart of the proposed algorithm. The key ingredients include:1.A novel topic identification scheme.2.Lateral expansion using novel topic incompatibility measure.3.Hierarchical expansion scheme using novel cluster size and topic size criteria.Display Omitted HighlightsIncorporation of topic identification into SOM learning could be beneficial to text categorization task.Both lateral and hierarchical expansion during SOM learning were achieved according to criteria based on identified topics.The produced text hierarchies outperformed contemporary approaches in quality and performance on text categorization. Self-organizing maps (SOM) have been applied on numerous data clustering and visualization tasks and received much attention on their success. One major shortage of classical SOM learning algorithm is the necessity of predefined map topology. Furthermore, hierarchical relationships among data are also difficult to be found. Several approaches have been devised to conquer these deficiencies. In this work, we propose a novel SOM learning algorithm which incorporates several text mining techniques in expanding the map both laterally and hierarchically. On training a set of text documents, the proposed algorithm will first cluster them using classical SOM algorithm. We then identify the topics of each cluster. These topics are then used to evaluate the criteria on expanding the map. The major characteristic of the proposed approach is to combine the learning process with text mining process and makes it suitable for automatic organization of text documents. We applied the algorithm on the Reuters-21578 dataset in text clustering and categorization tasks. Our method outperforms two comparing models in hierarchy quality according to users' evaluation. It also receives better F1-scores than two other models in text categorization task.

Social bookmarking Websites are popular nowadays for they provide platforms that are easy and clear to browse and organize Web pages. Users can add tags on Web pages to allow easy comprehension and retrieval of Web pages. However, tag spams could also be added to promote the opportunity of being referenced of a Web page, which is troublesome to users for accessing uninterested Web pages. In this work, we proposed a scheme to automatically detect such tag spams using a proposed text mining approach based on self-organizing map (SOM) model. We used SOM to find the associations among Web pages as well as tags. Such associations were then used to discover the relationships between Web pages and tags. Tag spams can then be detected according to such relationships. Experiments were conducted on a set of Web pages collected from a social bookmarking site and obtained promising result.

Highlights? We applied a density-based stream clustering method for mining Twitter data. ? The developed method can detect real-time and geospatial event features. ? Using the detection results can estimate the temporal and spatial impacts of events. ? Our method is well suited for awareness of large-scale events and risk management. Social networks have been regarded as a timely and cost-effective source of spatio-temporal information for many fields of application. However, while some research groups have successfully developed topic detection methods from the text streams for a while, and even some popular microblogging services such as Twitter did provide information of top trending topics for selection, it is still unable to fully support users for picking up all of the real-time event topics with a comprehensive spatio-temporal viewpoint to satisfy their information needs. This paper aims to investigate how microblogging social networks (i.e. Twitter) can be used as a reliable information source of emerging events by extracting their spatio-temporal features from the messages to enhance event awareness. In this work, we applied a density-based online clustering method for mining microblogging text streams, in order to obtain temporal and geospatial features of real-world events. By analyzing the events detected by our system, the temporal and spatial impacts of the emerging events can be estimated, for achieving the goals of situational awareness and risk management.

Highlights? We developed a fusion method to integrate text and photo types of location information. ? It combines image annotation and geographic text mining. ? Photos and texts are mapped to a unified semantic space for fusion. ? The approach can associate text and image types of documents with correct geographical zones. Due to the steady increase in the number of heterogeneous types of location information on the internet, it is hard to organize a complete overview of the geospatial information for the tasks of knowledge acquisition related to specific geographic locations. The text- and photo-types of geographical dataset contain numerous location data, such as location-based tourism information, therefore defining high dimensional spaces of attributes that are highly correlated. In this work, we utilized text- and photo-types of location information with a novel approach of information fusion that exploits effective image annotation and location based text-mining approaches to enhance identification of geographic location and spatial cognition. In this paper, we describe our feature extraction methods to annotating images, and utilizing text mining approach to analyze images and texts simultaneously, in order to carry out geospatial text mining and image classification tasks. Subsequently, photo-images and textual documents are projected to a unified feature space, in order to generate a co-constructed semantic space for information fusion. Also, we employed text mining approaches to classify documents into various categories based upon their geospatial features, with the aims to discovering relationships between documents and geographical zones. The experimental results show that the proposed method can effectively enhance the tasks of location based knowledge discovery.

With the increasing number of multilingual texts in the internet, multilingual text retrieval techniques have become an important research issue. However, the discovery of relationships between different languages remains an open problem. In this paper we propose a method, which applies the growing hierarchical self-organizing map (GHSOM) model, to discover knowledge from multilingual text documents. Multilingual parallel corpora were trained by the GHSOM to generate hierarchical feature maps. A discovery process is then applied on these maps to discover the relationships between documents of different languages. The relationships between keywords of different languages are also revealed. We conducted experiments on a set of Chinese—English bilingual parallel corpora to discover the relationships between documents of these languages. We also use such relationships to perform multilingual information retrieval tasks. The experimental results show that our multilingual text mining approach may capture conceptual relationships among documents as well as keywords written in different languages.

Traditional content-based image retrieval (CBIR) systems often fail to meet a user's need due to the 'semantic gap' between the extracted features of the systems and the user's query. The cause of the semantic gap is the failure of extracting real semantics from an image and the query. To extract semantics of images, however, is a difficult task. Most existing techniques apply some predefined semantic categories and assign the images to appropriate categories through some learning processes. Nevertheless, these techniques always need human intervention and rely on content-based features. In this paper we propose a novel approach to bridge the semantic gap which is the major deficiency of CBIR systems. We conquer the deficiency by extracting semantics of an image from the environmental texts around it. Since an image generally co-exists with accompanying texts in various formats, we may rely on such environmental texts to discover the semantics of the image. We apply a text mining process, which adopts the self-organizing map (SOM) learning algorithm as a kernel, on the environmental texts of an image to extract the semantic information from this image. Some implicit semantic information of the images can be discovered after the text mining process. We also define a semantic relevance measure to achieve the semantic-based image retrieval task. We performed experiments on a set of images which are collected from web pages and obtained promising results.

The research on automatic hypertext construction emerges rapidly in the last decade because there exists a urgent need to translate the gigantic amount of legacy documents into web pages. Unlike traditional 'flat' texts, a hypertext contains a number of navigational hyperlinks that point to some related hypertexts or locations of the same hypertext. Traditionally, these hyperlinks were constructed by the creators of the web pages with or without the help of some authoring tools. However, the gigantic amount of documents produced each day prevent from such manual construction. Thus an automatic hypertext construction method is necessary for content providers to efficiently produce adequate information that can be used by web surfers. Although most of the web pages contain a number of non-textual data such as images, sounds, and video clips, text data still contribute the major part of information about the pages. Therefore, it is not surprising that most of automatic hypertext construction methods inherit from traditional information retrieval research. In this work, we will propose a new automatic hypertext construction method based on a text mining approach. Our method applies the self-organizing map algorithm to cluster some at text documents in a training corpus and generate two maps. We then use these maps to identify the sources and destinations of some important hyperlinks within these training documents. The constructed hyperlinks are then inserted into the training documents to translate them into hypertext form. Such translated documents will form the new corpus. Incoming documents can also be translated into hypertext form and added to the corpus through the same approach. Our method had been tested on a set of at text documents collected from a newswire site. Although we only use Chinese text documents, our approach can be applied to any documents that can be transformed to a set of index terms.

Recently research on text mining has attracted lots of attention from both industrial and academic fields. Text mining concerns of discovering unknown patterns or knowledge from a large text repository. The problem is not easy to tackle due to the semi-structured or even unstructured nature of those texts under consideration. Many approaches have been devised for mining various kinds of knowledge from texts. One important aspect of text mining is on automatic text categorization, which assigns a text document to some predefined category if the document falls into the theme of the category. Traditionally the categories are arranged in hierarchical manner to achieve effective searching and indexing as well as easy comprehension for human beings. The determination of category themes and their hierarchical structures were most done by human experts. In this work, we developed an approach to automatically generate category themes and reveal the hierarchical structure among them. We also used the generated structure to categorize text documents. The document collection was trained by a self-organizing map to form two feature maps. These maps were then analyzed to obtain the category themes and their structure. Although the test corpus contains documents written in Chinese, the proposed approach can be applied to documents written in any language and such documents can be transformed into a list of separated terms.

This paper describes our work on developing a language-independent technique for discovery of implicit knowledge from multilingual information sources. Text mining has been gaining popularity in the knowledge discovery field, particularity with the increasing availability of digital documents in various languages from all around the world. However, currently most text mining tools mainly focus only on processing monolingual documents (particularly English documents): little attention has been paid to apply the techniques to handle the documents in Asian languages, and further extend the mining algorithms to support the aspects of multilingual information sources. In this work, we attempt to develop a language-neutral method to tackle the linguistics difficulties in the text mining process. Using a variation of automatic clustering techniques, which apply a neural net approach, namely the Self-Organizing Maps (SOM), we have conducted several experiments to uncover associated documents based on a Chinese corpus, Chinese-English bilingual parallel corpora, and a hybrid Chinese-English corpus. The experiments show some interesting results and a couple of potential paths for future work in the field of multilingual information discovery. Besides, this work is expected to act as a starting point for exploring the impacts on linguistics issues with the machine-learning approach to mining sensible linguistics elements from multilingual text collections.

In this work we utilize the social messages to construct an extensible event ontology model for learning the experiences and knowledge to cope with emerging real-world events. We develop a platform combining several text mining and social analysis algorithms to cooperate with our stream mining approach to detecting large-scale disastrous events from social messages, in order to achieve the aim of automatically constructing event ontology for emergency response First, we employ the developed event detection technique on Twitter social-messages to monitor the occurrence of emerging events, and record the development and evolution of detected events. Furthermore, we store the messages associated with the detected events in a repository. Through the developed algorithms for analyzing the content of social messages and ontology construction the event ontology can be established, allowing for developing relevant applications for prediction of possible evolution and impact evaluation of the events in the future immediately, in order to achieve the goals for early warning of disasters and risk management.

With the continuously increasing needs of location information for users around the world, applications of geospatial information have gained a lot of attention in both research and commercial organizations. Extraction of semantics from the image content for geospatial information seeking and knowledge discovery has been thus becoming a critical process. Unfortunately, the available geographic images may be blurred, either too light or too dark. It is therefore often hard to extract geographic features directly from images. In this paper, we describe our developed methods in applying local scale-invariant features and bag-of-keypoints techniques to annotating images, in order to carry out image categorization and geographic knowledge discovery tasks. First, local scale-invariant features are extracted from geographic images as representative geographic features. Subsequently, the bag-of-keypoints methods are used to construct a visual vocabulary and generate feature vectors to support image categorization and annotation. The annotated images are classified by using geographic nouns. The experimental results show that the proposed approach is sensible and can effectively enhance the tasks of geographic knowledge discovery.

Online microblogging services such as Twitter allow users to post very short messages related to everything ranging from mundane daily life routines to breaking news events. This phenomenon has changed the way for information acquisition. In this paper, we present an instinctive method with a time-decay function which corresponds to the natural propagation of social networks for clustering real-time text streams collected from Twitter. Compared to most previous studies, we follow natural cascading behaviors of event lifecycle to develop a self-adaptive clustering model for online event detection. Also, we construct an expandable similarity matrix which is capable of evaluating microblogging posts with incomplete semantic features. Experimental results show that the proposed method is a sensible solution to monitoring momentous real-time events and utilizing the text streams to facilitate the management of social networking data.

One of the basic human needs is to exchange information and socialize with each other. Online microblogging services such as Twitter allow users to post very short messages related to everything ranging from mundane daily life routines to breaking news events. A key challenging issue of mining such social messages is how to analyze the real-time distributed messages and extract significant features of them in a dynamic environment. In this work, we propose a novel term weighting method, called BursT, using sliding window techniques for weighting message streams. The experimental results show that our weighting technique has an outstanding performance to reflect the shifts of concept drift. The result of this work can be extended to perform a periodic feature extraction, and also be able to integrate other sophisticated clustering methods to enhance the efficiency for real-time event mining in social networks.

Social tags are annotations for Web pages collaboratively added by users. It will be much easier to understand the meaning of Web pages and classify them according to their tags. The precision in retrieving Web pages may also increase using such tags. Nowadays social tags are mostly annotated manually by users via social bookmarking Web sites. Such manual annotation process may produce diverse, redundant, and inconsistent tags. Besides, many tags which are inconsistent with their annotated Web pages exist and deteriorate the feasibility of social tags. In this work we will develop an automatic scheme to discover the associations between Web pages and social tags and apply such associations on applications of social tag spam detection. We applied a text mining approach based on self-organizing maps to find the relationships between Web pages and social tags. The disadvantages of manual annotation will be remedied through such relationships. The discovered associations were then used to identify social tag spams. Preliminary experiments show that the quality and usability of social tags were improved through this method.

Self-organizing map (SOM) learning algorithm has been widely applied in solving various tasks in pattern recognition, machine learning, and data mining, etc. Recently, it has been used to cluster documents and produced reasonable results. Traditional SOM algorithm learns from data using a fixed map. Approaches have been proposed to allow adaptable map structure. In this work, we propose a novel SOM learning algorithm that can expand the map laterally and hierarchically. The adaption of the map structure is based on topics identified from the underlying document clusters. This approach is different from traditional approaches which are typically driven by the data density of clusters. Preliminary experiment result suggested that the proposed algorithm outperforms other similar approaches.

With the increasing needs of location information, applications of geospatial information have gained a lot of attention in both research and commercial organizations Extraction of geospatial knowledge from the information content has been thus becoming a important process Among theses applications, a typical example is to discover relationships between various geospatial texts/data and specific locations In this paper, we describe a location based text mining approach using Artificial Neural Networks (ANN) to classify texts into various categories based on their geospatial features, with the aims to discovering relationships between documents and zones First, the collected documents were mapped to corresponding zones by the Adaptive Affinity Propagation (Adaptive AP) clustering techniques, then we performed framed maximize zones by means of Fuzzy ARTMAP (FAM) and Support Vector Machines (SVM) methods, allowing the results of relationships between documents and zones to be well presented Eventually, we compared our experimental results with that of baseline models using Self-organizing maps (SOM) and Learning Vector Quantization (LVQ) methods The preliminary results show that our platform framework has potential for geospatial knowledge discovery.

In this paper we describe our work on developing a novel technique for discovery of implicit knowledge about patents from multilingual patent information sources. In this work we developed a system platform to support locating similar and relevant multilingual patent documents. The platform was implemented using a multilingual vector space based on the latent semantic indexing (LSI) model, and utilizing collected professional Chinese-English parallel corpora for training the system model. These multilingual patent documents could then be mapped into the semantic vector space for evaluating their similarity by means of text clustering techniques. The preliminary results show that our platform framework has potential for retrieval and relatedness evaluation of multilingual patent documents.

In this paper, we develop a platform framework for categorization of cancer related abstracts using support vector machines (SVMs) based text categorization techniques with a one-against-all (OAA) learning algorithm for classification decisions. The corpora for the work were selected from the Website of PubMed database. By using information derived from PubMed literature source, including topics of breast cancer, cervical cancer, gastric cancer, lung cancer, rectum cancer and esophagus cancer, we randomly selected 6,000 medical abstracts for implementing our system and performing experiments. The experimental results show that the platform model has potentials for categorization of multiple cancer related literature texts.

The ability to automatically identify relationships between cancer diseases and external factors from medical records for supporting cancer diagnosis would be a valuable contribution in public health fields. Unfortunately, so far little attention has been paid on such a problem domain to developing effective solutions. In this work, we propose a prototype for automating the extraction of relationships between cancer diseases and potential factors from clinical records. We describe a scheme of the system prototype that integrates cancer ontology, and developed text mining techniques, covering self-organizing maps (SOM) algorithm as well as support vector machines (SVM) methods to carry out the system development. The results show that the integration of medical ontology and the text mining platforms is capable of extracting the potential patterns and re-categorize clinical records.

In this paper we discuss the implementation of the leading supervised and unsupervised approaches for multilingual text categorization. We selected support vector machines (SVM) and latent semantic indexing (LSI) techniques as representatives of supervised and unsupervised methods for system implementation, respectively. The preliminary results show that our platform models including both supervised and unsupervised learning methods have the potentials for multilingual text categorization

The WWW provides an ultimate source of information for all kinds of knowledge in various kinds of languages. There are emerging needs for searching documents in different languages, causing multilingual information retrieval an active research topic recently. The performance of such task depends on the degree of understanding for the relationships between different languages. Multilingual text mining aims at discovering interesting relationships between different languages. In this work, we applied the growing hierarchical self-organizing map model to cluster multilingual text documents and find the relationships between two languages. We use a set of parallel corpora to train the map and apply a discovering process to identify the semantic groups and hierarchical structures of keywords for these languages. The discovered knowledge can then be applied to tasks such as multilingual information retrieval and automatic multilingual thesaurus construction.

Topic maps provide a general, powerful, and user-oriented way to navigate the information resources under consideration in any specific domain. A topic map provides a uniform framework that not only identifies important subjects from an entity of information resources and specifies the resources that are semantically related to a subject, but also explores the relations among these subjects. When a user needs to find some specific information on a pool of information resources, he or she only needs to examine the topic maps of this pool, select the topic that seems interesting, and the topic maps will display the information resources that are related to this topic, as well as its related topics. The user will also recognize the relationships among these topics and the roles they play in such relationships. With the help of the topic maps, you no longer have to browse through a set of hyperlinked documents and hope that you may eventually reach the information you need in a finite amount of time, while knowing nothing about where to start. You also don’t have to gather some words and hope that they may perfectly symbolize the idea you’re interested in, and be well-conceived by a search engine to obtain reasonable result. Topic maps provide a way to navigate and organize information, as well as create and maintain knowledge in an infoglut. To construct a topic map for a set of information resources, human intervention is unavoidable at the present time. Human effort is needed in tasks such as selecting topics, identifying their occurrences, and revealing their associations. Such a need is acceptable only when the topic maps are used merely for navigation purposes and when the volume of the information resource is considerably small. However, a topic map should not only be a topic navigation map. The volume of the information resource under consideration is generally large enough to prevent the manual construction of topic maps. To expand the applicability of topic maps, some kind of automatic process should be involved during the construction of the maps. The degree of automation in such a construction process may vary for different users with different needs. One person may need only a friendly interface to automate the topic map authoring process, while another may try to automatically identify every component of a topic map for a set of information resources from the ground up. In this article, we recognize the importance of topic maps not only as a navigation tool but also as a desirable scheme for knowledge acquisition and representation. According to such recognition, we try to develop a scheme based on a proposed text-mining approach to automatically construct topic maps for a set of information resources. Our approach is the opposite of the navigation task performed by a topic map to obtain information. We extract knowledge from a corpus of documents to construct a topic map. Although currently the proposed approach cannot fully construct the topic maps automatically, our approach still seems promising in developing a fully automatic scheme for topic map construction.

The quantification of evaluating semantic relatedness among texts has been a challenging issue that pervades much of machine learning and natural language processing. This paper presents a hybrid approach of a text-mining technique for measuring semantic relatedness among texts. In this work we develop several text classifiers using support vector machines (SVM) method to supporting acquisition of relatedness among texts. First, we utilized our developed text mining algorithms, including text mining techniques based on classification of texts in several text collections. After that, we employ various SVM classifiers to deal with evaluation of relatedness of the target documents. The results indicate that this approach can also be fitted to other research work, such as information filtering, and recategorizing resulting documents of search engine queries.

The World Wide Web (WWW) has been recognized as the ultimate and unique source of information for information retrieval and knowledge discovery communities. Tremendous amount of knowledge are recorded using various types of media, producing enormous amount of web pages in the WWW. Retrieval of required information from the WWW is thus an arduous task. Different schemes for retrieving web pages have been used by the WWW community. One of the most widely used scheme is to traverse predefined web directories to reach a user's goal. These web directories are compiled or classified folders of web pages and are usually organized into hierarchical structures. The classification of web pages into proper directories and the organization of directory hierarchies are generally performed by human experts. In this work, we provide a corpus-based method that applies a kind of text mining techniques on a corpus of web pages to automatically create web directories and organize them into hierarchies. The method is based on the self-organizing map learning algorithm and requires no human intervention during the construction of web directories and hierarchies. The experiments show that our method can produce comprehensible and reasonable web directories and hierarchies.

This paper describes a new application of a text-mining algorithm to the text sources of bilingual corpora. In the past, the majority of the approaches applied to measuring semantic relatedness was based on edge counting methods through a semantic network, such as WordNet. It is not well suited for applications in specific domains in which the standard lexical knowledge bases are not available. In this work, we propose an alternative solution for acquisition of semantic relatedness from text corpora by means of a machine learning technique, namely the self-organizing maps. This paper presents a hybrid approach to discovering a concept-based feature map containing word clusters and document clusters from multilingual text collections. Using SOM-based automatic clustering techniques, we have conducted several experiments to uncover associated documents based on Chinese-English bilingual parallel corpora, and a hybrid Chinese-English corpus. In essence, this work provides a method for automatic text clustering, which resolves some of the language difficulties in concept discovery and categorization from multilingual text corpora.

In this paper we propose a novel method to bridge the 'semantic gap' between a user's information need and the image content. The semantic gap describes the major deficiency of content-based image retrieval (CBIR) systems which use visual features extracted from images to describe the images. We conquer the deficiency by extracting semantic of an image from the environmental texts around it. Since an image generally co-exists with accompany texts in various formats, we may rely on such environmental texts to discover the semantic of the image. A text mining approach based on self-organizing maps is used to extract the semantic of an image from its environmental texts. We performed experiments on a small set of images and obtained promising results.

Traditional content-based image retrieval (CBIR) systems often fail to fulfill a user’s need due to the ‘semantic gap’ existed between the extracted features of the systems and the user’s query. In this paper we propose a novel approach to bridge the semantic gap which is the major deficiency of CBIR systems. We conquer the deficiency by extracting semantics of an image from the environmental texts around it. We apply a text mining process, which adopts the self-organizing map (SOM) learning algorithm as a kernel, on the environmental texts of an image to extract the semantic information from this image. Some implicit semantic information of the images can be discovered after the text mining process. We also define a semantic relevance measure to achieve the semantic-based image retrieval task. We performed experiments on a set of images which are collected from web pages and obtained promising results.

Topic maps standard (ISO-13250) has been gradually recognized as an emerging standard for information exploration and knowledge organization in the web era. One advantage of topic maps is that they enable a user to navigate and access the documents he wants in an organized manner, rather than browsing through hyperlinks that are generally unstructured and often misleading. Nowadays, the topic maps are generally manually constructed by domain experts or users since the functionality and feasibility of automatically generated topic maps still remain unclear. In this work we propose a semi-automatic scheme to construct topic maps. We first apply a text mining process on a corpus of information resources to identify the topics and discover the relations among these topics. Necessary components of topic maps such as topics, topic types, topic occurrences, and topic associations may be automatically revealed by our method.

In this paper we propose a novel method to discover the semantics of an image within a web page and bridge the semantic gap between the image features and a user’s query. Since an image is always accompanied with some text segments which are closely related to the image, we propose that the semantics of the image may be discovered from such text segments through a data mining process on these texts. Based on such recognition, we applied a text mining process on the accompany texts of an image to discover the themes of this image, which constitute the semantics of this image. The self-organizing map algorithm is first applied to cluster a set of preprocessed web pages. Based on the clustering result, we design a theme identification process to identify a set of words which constitute the semantics of an image. We performed experiments on a small set of images and obtained promising results.

The paper describes a new application of a text-mining algorithm to the text sources of bilingual parallel corpora. The ultimate task, being undertaken in the context of a Chinese-English machine translation project, will be to develop a language-neutral method to discover similar documents from multilingual text collections. Using a variation of automatic clustering techniques which apply a neural net approach, namely the self-organizing maps (SOM), we have conducted several experiments to uncover associated documents based on Chinese-English bilingual parallel corpora, and a hybrid Chinese-English corpus. The experiments show some interesting results and a couple of potential ways for future work towards the field of multilingual information discovery. In addition, for exploring the impacts on linguistic issues with the machine learning approach to mining sensible linguistics elements from multilingual texts, we have examined the resulting term associations and text associations from the view of cross-lingual text similarity. To evaluate semantic relatedness of the mined bilingual texts, we applied a measure technique of semantic similarity in the resulting bilingual document clusters and word clusters. The paper presents algorithms that enable multilingual text mining based on the self-organizing map (SOM) for automatically grouping similar multilingual texts (i.e. Chinese and English texts), along with a means of measuring their semantic similarity to resolve the difficulties of syntactic and semantic ambiguity in multilingual information access.

In this work we developed a new automatic hypertext construction method based on a proposed text mining approach. Our method applies the self-organizing map algorithm to cluster some flat text documents in a training corpus and generate two maps. We then use these maps to identify the sources and destinations of some important hyperlinks within these training documents. The constructed hyperlinks are then inserted into the training documents to translate them into hypertext form. Such translated documents form the new corpus. Incoming documents can also be translated into hypertext form and added to the corpus through the same approach. Our method had been tested on a set of flat text documents collecting from several newswire sites. Although we only used Chinese text documents, our approach can be applied to any document that can be transformed to a set of indexed terms.

One important task for text data mining is automatic text categorization, which assigns a text document to some predefined category according to their correlations. Traditionally, these categories as well as the correlations among them are determined bp human experts. In this paper, we devised a novel approach to automatically generate categories. The self-organizing map model is used to generate two maps, namely the word cluster map and the document cluster map, in which a neuron represents a cluster of words and documents respectively. Our approach is to analyze the document cluster map to find centroids of some super-clusters. We also devised a method to select the category term from the word cluster map. The hierarchical structure of categories may be generated by recursively applying the same method. Text categorization is the natural consequence of such automatic category generation process.

Recently knowledge discovery and data mining in unstructured or semi-structured texts(text mining) has been attracted lots of attention from both commercial and research fields. One aspect of text mining is on automatic text categorization, which assigns a text document to some predefined category according to the correlation between the document and the category. Traditionally the categories are arranged in hierarchical manner to achieve effective searching and indexing as well as easy comprehension for human. The determination of categories and their hierarchical structures were most done by human experts. In this work, we developed an approach to automatically generate categories and reveal the hierarchical structure among them. We also used the generated structure to categorize text documents. The document collection is trained by a self-organizing map to form two feature maps. We then analyzed the two maps to obtain the categories and the structure among them. Although the corpus contains documents written in Chinese, the proposed approach can be applied to documents written in any language and such documents can be transformed into a list of separated terms.