Studying expert initial set and hard to map cases in automated code-to-architecture mappings

We study the mapping of software source code to architectural modules. Background: To evaluate techniques for performing automatic mapping of code-to-architecture, a ground truth mapping, often provided by an expert, is needed. From this ground truth, techniques use an initial set of mapped source code as a starting point. The size and composition of this set affect the techniques’ performance, and to make comparisons, random sizes and compositions are used. However, while randomness will give a baseline for comparison, it is not likely that a human expert would compose an initial set on random to map source code. We are interested in letting an expert create an initial set based on their experience with the system and study how this affects how a technique performs. Also, previous research has shown that when comparing an automatic mapping with the ground truth mappings, human experts often accept the automated mappings and, if not, point to the need for refactoring the source code. We want to study this phenomenon further. Audience: Researchers and developers of tools in the area of architecture conformance. The system expert can gain valuable insights into where the source code needs to be refactored. Aim: We hypothesize that an initial set assigned by an expert performs better than a random initial set of similar size and that an expert will agree upon or find opportunities for refactoring in a majority of cases where the automatic mapping and expert mapping disagrees. Method: The initial set will be extracted from an interview with the expert. Then the performance (precision and recall f1 score) will be compared to mappings starting from random initial sets and using an automatic technique. We will also use our tool to find the cases where the automatic and human mapping disagrees and then let the expert review these cases. Results: We expect to find a difference when performance is compared. We expect the expert review to reveal source code that should be rema