Recommendation System for Product Test Failures Using BERT

Historical failure records can provide insights to investigate if a similar situation occurred during the troubleshooting process in software. However, in the era of information explosion, massive amounts of data make it unrealistic to rely solely on manual inspection of root causes, not to mention mapping similar records. With the ongoing development and breakthroughs of Natural Language Processing (NLP), we propose an end-to-end recommendation system that can instantly generate a list of similar records given a new raw failure record. The system consists of three stages: 1) general and tailored pre-processing of raw failure records; 2) information retrieval; 3) information re-ranking. In the process of model selection, we undertake a thorough exploration of both frequency-based models and language models. To mitigate issues stemming from imbalances in the available labeled data, we propose an updated Recall@K metric that utilizes an adaptive K. We also develop a multi-stage training pipeline to deal with limited labeled data and investigate how different strategies affect performance. Our comprehensive experiments demonstrate that our two-stage BERT model, fine-tuned on extra domain data, achieves the best score over the baseline models.
Part of ISBN 9789897586712QC 20231229