Benchmarking Numerical Multiobjective Optimizers Revisited

Dimo Brockhoff 1, * Thanh-Do Tran 1 Nikolaus Hansen 2
* Corresponding author
1 DOLPHIN - Parallel Cooperative Multi-criteria Optimization
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
2 TAO - Machine Learning and Optimisation
CNRS - Centre National de la Recherche Scientifique : UMR8623, Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, LRI - Laboratoire de Recherche en Informatique
Abstract : Algorithm benchmarking plays a vital role in designing new optimization algorithms and in recommending efficient and robust algorithms for practical purposes. So far, two main approaches have been used to compare algorithms in the evolutionary multiobjective optimization (EMO) field: (i) displaying empirical attainment functions and (ii) reporting statistics on quality indicator values. Most of the time, EMO benchmarking studies compare algorithms for fixed and often arbitrary budgets of function evaluations although the algorithms are anytime optimizers. Instead, we propose to transfer and adapt standard benchmarking techniques from the single-objective optimization and classical derivative-free optimization community to the field of EMO. Reporting target-based runlengths allows to compare algorithms with varying numbers of function evaluations quantitatively. Displaying data profiles can aggregate performance information over different test functions, problem difficulties, and quality indicators. We apply this approach to compare three common algorithms on a new test function suite derived from the well-known single-objective BBOB functions. The focus thereby lies less on gaining insights into the algorithms but more on showcasing the concepts and on what can be gained over current benchmarking approaches.
Complete list of metadatas

Cited literature [22 references]  Display  Hide  Download
Contributor : Dimo Brockhoff <>
Submitted on : Wednesday, April 29, 2015 - 12:21:02 AM
Last modification on : Friday, March 22, 2019 - 1:34:56 AM
Long-term archiving on : Monday, September 14, 2015 - 2:57:12 PM


Files produced by the author(s)



Dimo Brockhoff, Thanh-Do Tran, Nikolaus Hansen. Benchmarking Numerical Multiobjective Optimizers Revisited. Genetic and Evolutionary Computation Conference (GECCO 2015), Jul 2015, Madrid, Spain. pp.639-646, ⟨10.1145/2739480.2754777⟩. ⟨hal-01146741⟩



Record views


Files downloads