Automating GUI Testing with Image-Based Deep Reinforcement Learning

Users interact with modern applications and devices through graphical user interfaces (GUIs). To ensure intuitive and easy usability, the GUIs need to be tested, where developers aim at finding possible bugs and inconsistent functionality. Manual GUI testing requires time and effort, and thus, its efficiency can be improved with automation. Conventional automation tools for GUI testing reduce the burden of manual testing but also introduce challenges in the maintenance of test cases. In order to overcome these issues, we propose a deep-reinforcement-learning-based (DRL) solution for automated and adaptive GUI testing. Specifically, we propose and evaluate the performance of an image-based DRL solution. We adapt the asynchronous advantage actor-critic (A3C) algorithm to GUI testing inspired by how a human uses a GUI. We feed screenshots of the GUI as the input and let the algorithm decide how to interact with GUI components. We observe that our solution can achieve up to six times higher exploration efficiency compared to selected baseline algorithms. Moreover, our solution is more efficient than inexperienced human users and almost as efficient as an experienced human user in our experimental GUI testing scenario. For these reasons, image-based DRL exploration can be considered as a viable GUI testing method.