Search

The main difficulty arising in the process of automating the retrieval of objects from heterogeneous distributed information bases of an enterprise is the problem of unification of disparate content presented from different points of view and in the context of different paradigms for organizing data storage. The article presents the formulation of the problem of developing graphematic analysis for the purpose of recognizing images of technical documentation and converting graphic information into a machine-readable form, the mechanisms for removing stop words, stemming, lemmatization necessary for solving the problem are described in detail, and an algorithm for searching text structures using templates is developed. The article proposes the implementation of the graphematic analysis algorithm as the first module in the automatic processing of texts in natural language, which makes it possible to parcel out semantically significant constructions from semi-structured resources using special graphematic descriptors. The proposed implementation makes it possible to parcel out such complex structures in natural language, such as, for example, direct speech, to detect and replace abbreviations and abbreviations. [ABSTRACT FROM AUTHOR]

A technique has been developed for experimental verification of software that implements the algorithm for graphematic analysis and preprocessing of text resources. This software is used to solve the problem of automated mining analysis of poorly structured data. It is designed to extract semantically significant constructs from poorly structured resources, which is achieved by transforming text using a number of auxiliary algorithms and classifying the data obtained using machine learning algorithms in Python. At the first stage of the developed algorithm, abbreviations and acronyms are searched in a text file. After searching for abbreviations and acronyms using templates, a search is performed for graphematic descriptors, namely name, email and URL, phone number and date. At the final stage of the developed algorithm, the boundaries of sentences and direct speech are distinguished. [ABSTRACT FROM AUTHOR]