Search

The process of learning is typical for the most assistive and autonomous robots. Considering the learning as a complex process, the issue of the data labeling is in the focus of this paper, because of the labeling noise effect on the learning results. Distributed redundant labeling is discussed in recent works, yet, no known publications contain the process description in the focus of the distributed decentralized processes, although the manner of distributed implementation of the various areas of the robotics is relevant now in the angle of fog- and edge-robotics. The current paper is devoted to the question of the redundant distributed labeling under the conditions of the decentralized data storing, which is quite relevant in the fog- and edge- robots environments. Two main approaches of the distributed monitoring of the labeling experts groups are considered and compared, as well as some experimental research of the redundant labeling results has been conducted with the results presented.

Extracting scientific facts from unstructured text is difficult due to challenges specific to the complexity of the scientific named entities and relations to be extracted. This problem is well illustrated through the extraction of polymer names and their properties. Even in the cases where the property is a temperature, identifying the polymer name associated with the temperature may require expertise due to the use of complicated naming conventions and by the fact that new polymer names are being “introduced” into the lexicon as polymer science advances. While domain-specific machine learning toolkits exist that address these challenges, perhaps the greatest challenge is the lack of—time-consuming, error-prone and costly—labeled data to train these machine learning models. This work repurposes Snorkel, a data programming tool, in a novel approach as a way to identify sentences that contain the relation of interest in order to generate training data, and as a first step towards extracting the entities themselves. By achieving 94% recall and an F1 score of 0.92, compared to human experts who achieve 77% recall and an F1 score of 0.87, we show that our system captures sentences missed by both a state-of-the-art domain-aware natural language processing toolkit and human expert labelers. We also demonstrate the importance of identifying the complex sentences prior to extraction by comparing our application to the natural language processing toolkit.

Understanding what a person is asking via a question is one of the first steps that humans use to find the corresponding answer. The same is true for Question Answering (QA) systems. Hence, the quality of the expected answer type classifier (EAT) has a direct influence on QA quality. Many research papers are aiming at improving short text classification quality, however, there is a lack of focus on the impact of training data characteristics on the classification quality as well as effective reuse of datasets through their augmentation and combination. In this work, we propose an approach of analyzing and improving the EAT classification quality via a combination of existing QA datasets. We provide 4 new question classification datasets based on several well-known QA datasets as well as the approach to unify its class taxonomy. We made a sufficient amount of experiments to demonstrate several valuable insights related to the impact of training data characteristics on the classification quality. Additionally, an embedding-based approach for automatic data labeling error detection is demonstrated.

In this paper the issue of labeling noise is considered. Data labeling is the integral stage of the most part of the machine learning projects, so the problem spotlighted in the paper is quite topical.

Training dataset forming is quite labor intensive and frequently is of high costs. Also the cost overheads depend on the policies of the service, which implements the crowdsourcing approach to data labeling. In this paper a new peer-to-peer data labeling platform concept is presented, as well as the framework of the decentralized labeling approach is described briefly. The architecture proposed allows to avoid the intermediary labeling service and to perform the crowdsourcing-based data labeling by the computational facilities of users involved. Besides, the additional consensus procedure improves the quality of the labeled data by means of the voting procedure.

Labeling is a cornerstone of supervised machine learning. However, in industrial applications, data is often not labeled, which complicates using this data for machine learning. Although there are well-established labeling techniques such as crowdsourcing, active learning, and semi-supervised learning, these still do not provide accurate and reliable labels for every machine learning use case in the industry. In this context, the industry still relies heavily on manually annotating and labeling their data. This study investigates the challenges that companies experience when annotating and labeling their data. We performed a case study using a semi-structured interview with data scientists at two companies to explore their problems when labeling and annotating their data. This paper provides two contributions. We identify industry challenges in the labeling process, and then we propose mitigation strategies for these challenges.

The generating of training datasets for machine learning projects is a topical problem. The cost of dataset formation can be considerably high, yet there is no guarantee of an acceptable quality of prepared data. The important issue of dataset generation is the labeling noise. The main causes of this phenomena are: expert errors, information insufficiency, subjective factors and so on. Labeling noise affects the learning stage of a neuronet and so increases the number of errors during the one’s functioning. In the current paper the technique to decrease the labeling noise level is proposed. It is based on the principals of the distributed ledger technology. While there is a possibility to decrease the labeling errors number, the services integration on the basis of distributed ledger allows to improve the efficiency of dataset forming.

Because machine learning would benefit from reduced data requirements, some prior work has proposed using humans not just to label data, but also to explain those labels. To characterize the evidence humans might want to provide, we conducted a user study and a data experiment. In the user study, 75 participants provided classification labels for 20 photos, justifying those labels with free-text explanations. Explanations frequently referenced concepts (objects and attributes) in the image, yet 26% of explanations invoked concepts not in the image. Boolean logic was common in implicit form, but was rarely explicit. In a follow-up experiment on the Visual Genome dataset, we found that some concepts could be partially defined through their relationship to frequently co-occurring concepts, rather than only through labeling.

Multi-instance multi-label learning (MIML) is a framework in machine learning in which each object is represented by multiple instances and associated with multiple labels. This relatively new approach has achieved success in various applications, particularly those involving learning from complex objects. Because of the complexity of MIML, the cost of data labeling increases drastically along with the improvement of the model performance. In this paper, we introduce a MIML active learning approach to reduce the labeling costs of MIML data without compromising the model performance. Based on a query strategy, we select and request from the Oracle the label set of the most informative object. Our approach is formulated in a pool-based scenario and uses Miml-\(k\) nn as the base classifier. This classifier for MIML is based on the \(k\)-Nearest Neighbor algorithm and has achieved superior performance in different data domains. We proposed novel query strategies and also implemented previously used query strategies for MIML learning. Finally, we conducted an experimental evaluation on various benchmark datasets. We demonstrate that these approaches can achieve significantly improved results than without active selection for all datasets on various evaluation criteria.

Recent successes in visual recognition can be primarily attributed to feature representation, learning algorithms, and the ever-increasing size of labeled training data. Extensive research has been devoted to the first two, but much less attention has been paid to the third. Due to the high cost of manual data labeling, the size of recent efforts such as ImageNet is still relatively small in respect to daily applications. In this work, we mainly focus on how to automatically generate identifying image data for a given visual concept on a vast scale. With the generated image data, we can train a robust recognition model for the given concept. We evaluate the proposed webly supervised approach on the benchmark Pascal VOC 2007 dataset and the results demonstrates the superiority of our method over many other state-of-the-art methods in image data collection.

This paper introduces the work being carried out in an ongoing PhD research focused on the detection of the learners’ affective states by combining different available sources (from physiological sensors to keystroke analysis). Different data mining algorithms and data labeling techniques have been used generating 735 prediction models. Results so far show that predictive models on affective state detection from multimodal-based approaches provide better accuracy rates than single-based.