Evaluating different i*-based approaches for selecting functional requirements while balancing and optimizing non-functional requirements: A controlled experiment

Context: A relevant question in requirements engineering is which set of functional requirements (FR) to prioritize and implement, while keeping non-functional requirements (NFR) balanced and optimized. Objective: We aim to provide empirical evidence that requirement engineers may perform better at the task of selecting FRs while optimizing and balancing NFRs using an alternative (automated) i* post-processed model, compared to the original i* model. Method: We performed a controlled experiment, designed to compare the original i* graphical notation, with our post-processed i* visualizations based on Pareto efficiency (a tabular and a radar chart visualization). Our experiment consisted of solving different exercises of various complexity for selecting FRs while balancing NFR. We considered the efficiency (time spent to correctly answer exercises), and the effectiveness (regarding time: time spent to solve exercises, independent of correctness; and regarding correctness of the answer, independent of time). Results: The efficiency analysis shows it is 3.51 times more likely to solve exercises correctly with our tabular and radar chart visualizations than with i*. Actually, i* was the most time-consuming (effectiveness regarding time), had a lower number of correct answers (effectiveness regarding correctness), and was affected by complexity. Visual or textual preference of the subjects had no effect on the score. Beginners took more time to solve exercises than experts if i* is used (no distinction if our Pareto-based visualizations are used). Conclusion: For complex model instances, the Pareto front based tabular visualization results in more correct answers, compared to radar chart visualization. When we consider effectiveness regarding time, the i* graphical notation is the most time consuming visualization, independent of the complexity of the exercise. Finally, regarding efficiency, subjects consume less time when using radar chart visualization than tabular visualization, and even more so compared to the original i* graphical notation. Sven Casteleyn is funded under the Ramón y Cajal Program of the Spanish Government, grant number RYC-2014-16606. This work has been partially supported by the Publi@City project (TIN2016-78103-C2-2-R) from the Spanish Ministry of Economy and Competitiveness.