Query Strategies, Assemble! Active Learning with Expert Advice for Low-resource Natural Language Processing

Active learning plays an important role in low-resource scenarios, i.e., when only a small amount of annotated instances is available. However, one does not know what is the best active learning strategy before actually testing a handful of strategies on a labeled set, which might not be viable in a real world low-resource scenario. Instead, it would be desirable to dynamically obtain the results from the best strategy on a given scenario, while using as little annotated resources as possible.In this paper, we present a novel application of prediction with expert advice to combine different query strategies as experts, giving a greater weight to those which select the most useful instances. We evaluated our approach in two Natural Language Processing (NLP) tasks: Part-of-Speech tagging (for English) and Named Entity Recognition (for Portuguese). Results show that our solution keeps up with the results of the best strategy in each scenario, nearly reaching fully supervised performance with only half of the annotated data.