Search

The performance assessment of multi-objective optimization algorithms involves a user-preference-based selection of a single quality indicator used as a performance measure. A single quality indicator maps the approximation set (i.e., high-dimensional data) into a real value (i.e, one-dimensional data). However, it is well known that the selection of the quality indicator can have a huge impact on the benchmarking conclusions. This invites researchers to present only results for quality indicators that are in favor of the desired algorithm, or performing bias performance assessment. To go beyond this, we proposed a novel ranking scheme that reduces the bias in the user-preference selection by comparing the high-dimensional data of approximations sets and consequently provides more robust statistical results. The selection of a quality indicator is only required in cases when high-dimensional distributions of the approximation sets differ. By performing such analyses, experimental results show that the cases affected by the user-preference selection are reduced.

When designing a benchmark problem set, it is important to create a set of benchmark problems that are a good generalization of the set of all possible problems. One possible way of easing this difficult task is by using artificially generated problems. In this paper, one such single-objective continuous problem generation approach is analyzed and compared with the COCO benchmark problem set, a well know problem set for benchmarking numerical optimization algorithms. Using Exploratory Landscape Analysis and Singular Value Decomposition, we show that such representations allow us to further explore the relations between the problems by applying visualization and correlation analysis techniques, with the goal of decreasing the bias in benchmark problem assessment., To appear in the Proceedings of Genetic and Evolutionary Computation Conference Companion (GECCO 2021), ACM

The performance assessment of multi-objective optimization algorithms is a crucial task for investigating their behaviour. However, the selected quality indicators and statistical techniques used in comparison studies can have huge impact on the study results. A quality indicator transforms high-dimensional data (an approximation set) into one-dimensional data (a quality indicator), followed by a potential loss of high-dimensional information concerning the transformation. Comparison approaches typically involve a single quality indicator or an ensemble of quality indicators to address more quality criteria, which are predefined by the user. To provide more robust benchmarking for multi-objective optimization, we extended the DSCTool with three approaches that are ensembles of quality indicators and one novel approach that compare the high-dimensional distributions of the approximation sets and reduces the users' preference in the selection of quality indicators. The approaches are provided as web services for robust ranking and hypothesis testing, including a proper selection of an omnibus statistical test and post-hoc tests if needed.

Understanding of exploration and exploitation powers of meta-heuristic stochastic optimization algorithms is very important for algorithm developers. For this reason, we have recently proposed an approach for making a statistical comparison of meta-heuristic stochastic optimization algorithms according to the distribution of the solutions in the search space, which is also presented in this paper. Its main contribution is the support to identify exploration and exploitation powers of the compared algorithms. This is especially important when dealing with multimodal search spaces, which consist of many local optima with similar values, and large-scale continuous optimization problems, where it is hard to understand the reasons for the differences in performances. Experimental results showed that our recently proposed approach gives very promising results.

The GECCO Workshop on Real-Parameter Black-Box Optimization Benchmarking Series is a series of benchmarking workshops held every year since 2009 that evaluates the performance of new optimization algorithms. Originally, the workshop organizers provided results for every year the workshop took place. In this article, we directly compare algorithms from every workshop held from 2009 to 2018. We analyze the compared algorithms on the noiseless benchmark function set using the competition's official empirical approach, and with a statistical approach called Deep Statistical Comparison. Our goal is to investigate how algorithm performance has evolved throughout the years, and to show differences between empirical and statistical approaches to evaluating results.

In this paper a recently proposed approach for making a statistical comparison of meta-heuristic stochastic optimization algorithms is presented. The main contribution of this approach is that the ranking scheme is based on the whole distribution, instead of using only one statistic to describe the distribution, such as average or median. Experimental results showed that our approach gives more robust results compared to state-of-the-art approaches in case when the results are affected by outliers or by statistical insignificant differences that could exist between data values.