Investigating the applicability of execution tracing techniques for root causing randomness-related flaky tests in Python

Regression testing is an essential part of developing and maintaining software. It helps verify that changes to the software have not introduced any new bugs, and that the functionality still works as intended. However, for this verification to be valid, the executed tests must be assumed to be deterministic, i.e. produce the same output under the same circumstances. Unfortunately, this is not always the case. A test that exhibits non-deterministic behavior is said to be flaky. Flaky tests can severely inhibit the benefits of regression testing, as developers must figure out whether a failing test is due to a bug in the system under test (SUT) or test flakiness. Moreover, the non-deterministic nature of flaky tests poses several problems. Not only are the failures difficult to reproduce and debug, but developers are more likely to ignore the outcome of flaky tests, potentially leading to overlooked bugs in the SUT. The aim of this thesis was to investigate the applicability of execution tracing techniques as a means of providing root cause analysis for flaky tests in the randomness and network categories. This involved reproducing and studying flakiness, as well as implementing and evaluating a prototype with the ability to analyze runtime behavior in flaky tests. To gain a better understanding of reproducibility and common traits among flaky tests in the selected categories, a pre-study was conducted. Based on the outcome of the pre-study and findings in related literature, the network category was dropped entirely, and two techniques were chosen to be implemented. The implementation process resulted in the FlakyPy tool, a plugin for pytest that provides root cause analysis aimed at randomness flakiness. When run against a dataset of 22 flaky tests, the tool was able to identify potential root causes in 15 of these. This serves as an indication that execution tracing has the potential of detecting possible root causes in flaky randomness tests in Python. However, m